Protobuf is the worst serialization format and schema IDL, except for all the others.
The truth is, a lot of these critiques are valid. You could redesign protobuf from the ground up and improve it greatly. However, I think this article is bad. Firstly, it repeatedly implies that protobuf was designed by amateurs because of its design pitfalls, ignoring the fact that it's also clear that protobuf grew organically into what it is today rather than deliberately. Secondly, I do not feel it is giving enough credit to the idea that protobuf simply was not designed to be elegant in the first place. It's not a piece of modern art. It is not a Ferrari. It is a Ford F150.
What protobuf does have right is the stuff that's boring; it has a ton of language bindings that are mostly pretty efficient and complete. I'm not going to say it's all perfect, because it's not. But protobuf has an ecosystem that, say, Cap'n Proto does not. This matters more than the fact that protobuf is inelegant. Protobuf may be inelegant, but it does in fact get the job done for many of the use cases.
In my opinion, protobuf is actually good enough. It's not that ugly. Most programming languages you would use it from are more inelegant and incongruent than it is. I don't feel like modelling data or APIs in it is overly limited by this. Etc, etc. That said, if you want to make the most of protobuf, you'll want to make use of more than just protoc, and that's the benefit of having a rich ecosystem: there are plenty of good tools and libraries built around it. I'll admit that outside of Google, it's a bit harder to get going, but my experiences on side projects have made me feel that protobuf outside of Google is still a great idea. It is, in my opinion, significantly less of a pain in the ass than using OpenAPI based solutions or GraphQL, and much more flexible than language-specific tools like tRPC.
For a selection of what the ecosystem has to offer, I think this list is a good start.
I think this kind of gets to the point of what protobuf is, and what a lot of tech is like at Google in general. I spent a lot of my career believing elegance and expressiveness was so important to strive for, even if we as engineers often fall short.
But tech at Google tends to be pragmatic in the specific way that protobuf is. It's not perfect, it doesn't fit neatly into a grand ideology, but it is (at least within Google itself) simple enough, easy enough to understand, portable and fit for purpose. In a similar way to bazel, it's full of components worthy of criticism, but those fixes get made when they become the ecosystem's most pressing issues, and not before.
> Protobuf is the worst serialization format and schema IDL, except for all the others.
I don't even feel this way. I can complain for days about most technical things, including things Google has made (e.g. Golang), but not protobufs. They're just good. They don't totally replace JSON of course, but for the intended use case of internal distributed systems, they're solid.
JSON is good for dumps of your data, for humans reading it, and for interoperability without the schema - On your backends, you should always serialized to proto because it's darn more efficient.
If it's a startup or a personal project with no special performance requirements, chance of me using protobufs for anything is very slim. Human readability aside, just bothering to set up the serdes and other boilerplate is too low on my list of priorities. Makes more sense at medium scale.
Slapping it all into gRPC is less boilerplate than whatever else you'd be doing (unless you're doing GraphQL or something). I'd always default to doing that (or thrift, honestly, but same difference) unless there were a particular reason to use something like a manual API.
gRPC requires HTTP/2, which is a nontrivial added requirement. Prevents it from running on Heroku and ironically Google AppEngine, and wherever it does work, load balancers will probably need special setup for it. Even Kubernetes. You can translate it to/from JSON, but that defeats the purpose.
> gRPC requires HTTP/2, which is a nontrivial added requirement. Prevents it from running on Heroku and ironically Google AppEngine
Then use thrift, which does plain http. But most of the design decisions in the article are the same there as well. Turns out there's really only one way to do this.
Oh, gRPC over JSON still gives you the advantage of a well-defined API. But I use OpenAPI aka Swagger for that because it's a much thinner layer, especially if for whatever reason I need to handle HTTP GET (callbacks/webhooks) in addition to my typical HTTP POST JSON.
It's also possible to use JSON directly with gRPC/DRPC/Connect, so you can take advantage of a Protobuf IDL and codegen with a bit less of the logistical complexity.
Note that a solution does not (and should not) solve every problem. That does not make it bad. Very few (read: NO) solutions can solve every problem.
I've known this forever, but when my company started doing outreach to customers, the point was really driven home.
Find a problem. Solve it. That is it. If the solution is popular enough it will pick up momentum. There will always be naysayers. It might not be the best solution, or even the right one, but it IS a solution, which is better than what others are doing.
Think something is bad? Fix it. Submit a PR if it is open source. Don't complain for the sake of complaining unless you are willing to show how it should be done.
Note that I'm not one of those open source jerk holes, however I do recognize that unless I can show I can do better, my opinions are meaningless.
The truth is, a lot of these critiques are valid. You could redesign protobuf from the ground up and improve it greatly. However, I think this article is bad. Firstly, it repeatedly implies that protobuf was designed by amateurs because of its design pitfalls, ignoring the fact that it's also clear that protobuf grew organically into what it is today rather than deliberately. Secondly, I do not feel it is giving enough credit to the idea that protobuf simply was not designed to be elegant in the first place. It's not a piece of modern art. It is not a Ferrari. It is a Ford F150.
What protobuf does have right is the stuff that's boring; it has a ton of language bindings that are mostly pretty efficient and complete. I'm not going to say it's all perfect, because it's not. But protobuf has an ecosystem that, say, Cap'n Proto does not. This matters more than the fact that protobuf is inelegant. Protobuf may be inelegant, but it does in fact get the job done for many of the use cases.
In my opinion, protobuf is actually good enough. It's not that ugly. Most programming languages you would use it from are more inelegant and incongruent than it is. I don't feel like modelling data or APIs in it is overly limited by this. Etc, etc. That said, if you want to make the most of protobuf, you'll want to make use of more than just protoc, and that's the benefit of having a rich ecosystem: there are plenty of good tools and libraries built around it. I'll admit that outside of Google, it's a bit harder to get going, but my experiences on side projects have made me feel that protobuf outside of Google is still a great idea. It is, in my opinion, significantly less of a pain in the ass than using OpenAPI based solutions or GraphQL, and much more flexible than language-specific tools like tRPC.
For a selection of what the ecosystem has to offer, I think this list is a good start.
https://github.com/grpc-ecosystem/awesome-grpc#protocol-buff...