The TypeScript Web Protobuf Environment in 2023Published on 10 February 2023
Another title of this post would be: for protocol buffers in web clients, which TypeScript protocol buffer libraries are well-documented, and which ones are badly documented?
In this post, I’ll talk about the mess that’s the open source TypeScript protocol buffer ecosystem. For historical reasons, based on various libraries that were released when others didn’t exist, there are some libraries that work with each other, some that are outdated, etc. In general, this happens quite commonly in the open source ecosystem, and I hope this post helps others.
The TLDR, if you want to use protocol buffers with the browser:
- Do NOT use
My recommendations don’t even have to do with the functionality necessarily; they’re about the documentation. Note that documentation doesn’t just mean READMEs but also code examples, which are notably missing from the ones that I suggest not to use.
One disclaimer: perhaps for other side use cases those libraries are well-documented. For the particular use case of web RPCs, they are lacking.
The rest of the post is mostly an explanation of my findings of the current state of TypeScript protobuf tools. If you just want to know which tools to use, trust me on the TLDR, and skip the rest of the post.
I wanted to make a simple demo of a TypeScript React client connecting to a TypeScript Node backend.
I used to work at Google for four years, and like everyone who’s worked at Google (there’s even a derisive term “proto pusher” to sometimes the mindless work of protos), I’ve used protocol buffers extensively. I’m not claiming to be the world’s foremost expert on protocol buffers or that I should be able to understand anything about protos instantly, but I just want to make the point: I’ve used them to set up web flows, data pipelines, clients and servers from scratch, etc. With a foundational knowledge of protobufs and how they’re transmitted, I should, in theory, be able to find some documentation and set up a basic “Hello World” RPC quickly.
Of course, documentation is different inside and outside of Google, but that’s the point of this post. Given the wild environment of various solutions created at various times, I hope to provide an explanation of what works (until this post becomes outdated, of course).
In short: after spending five hours desperately searching for documentation to
use the combination I said not to use (which have higher number of stars on
GitHub), I set up a working end-to-end connection in 20 minutes using the
combination that I said to use (
protobuf-ts) because of good
Points of confusion
You have protocol buffers, which define a contract for entities that you want to work with. Now you want to generate Typescript types from those protos.
As far as I can tell, there exist two main open source options for type generation:
ts-proto(has more stars)
I think there’s also now TypeScript generation built into the official
but again, the documentation in this world is so obfuscated that I only figured
that out from reading
grpc/grpc-web example code five directories deep in the
protobuf-ts is the clear winner here, simply because of
MANUAL.md , while quite long, has stellar examples for use cases on both
the client- and server- sides. On top of that, there is a whole folder of
actual code examples,
ts-proto, from what I’ve seen in some responses to the issues, asks you
read their integration
but more difficult than parsing real code examples).
The author of
protobuf-ts has written a summary of the differences between
these two libraries,
although to be
honest, my currently visualized use case is basic enough that none of these will
likely matter to me. As someone newly figuring out the integrations in the
TypeScript protobuf world, I can recommend
protobuf-ts solely due to its
Client-side GRPC Client
There are two identically named repositories:
The former is created by Google, and the second by another company. However, the
second one is in maintenance mode with the recommendation to use the official
one, which wasn’t even posted on the repo until I made a
(the maintainer had been warning people on the repo’s issues).
For technical reasons (Google it if you’re curious), using GRPC in the browser requires
a proxy. The
improbable-eng/grpc-web library requires their own Go binary as a
proxy, which, upon realizing this, I immediately surmised that it would be
a nightmare to figure out how to deploy some random Go proxy if I
wanted to combine it with other things like AWS Lambda, etc. As expected, the
documentation for the proxy
is scant. I didn’t think that the documentation for deploying
would be promising either, and as expected, there’s little guidance there,
except for a few GitHub issues.
grpc/grpc-web suggests using Envoy
as a proxy, which is quite well-known, well-maintained, and well-documented.
It has Docker support, and is, in general, more well-vetted.
I initially tried the following combination:
improbable-eng/grpc-web. Both seemed popular. After getting a semi-working example using these two libraries, I then wanted to try the
ts-proto and official
grpc/grpc-web combinations (moving off the
improbable-eng due to the aforementioned proxy). From the supported options
ts-proto, it supports
improbable-eng/grpc-web, but not the official Google
It seems strange that
ts-proto wouldn’t integrate with the official
Google library. Or maybe it does? But that’s the point — perhaps it does support it, but
the documentation isn’t there. I couldn’t find any code examples, any flag
documentation, any references, and I couldn’t figure out how to get the types to
work on my own by digging into the generated types (admittedly due to my own
lack of knowledge but hey, that’s the point of Hello World examples).
And if it is really true that
ts-proto doesn’t work with the official,
Google-released version of
grpc/grpc-web, then it suggests that the tool may
be outdated. Perhaps that version
grpc/grpc-web didn’t exist then, but in any
case, the documentation should clarify this. The only reference I found to this
was this answer from the maintainer
about how the library predates gRPC, if I understood the post correctly.
For writing a TypeScript Node server, there does exist decent documentation for
version of the official tutorial.
protobuf-ts here worked fine. Ultimately, I only ran into
problems with client-side web RPC typing. Nonetheless, you would want to use
the same generator for both the server and client side.
In conclusion, documentation. But also, if you are in a similar boat to me and want to get started with a basic hello world, TypeScript gRPC web client and server, I suggest using the two libraries I talked about.
Finally, I want to say that this post does not mean to detract from the work of these open source contributors. Each of these libraries, at some point in time, had their crucial role in the ecosystem.