If you are on GCP I think the choice is simple, use Cloud Pub/Sub. Extremely simple, extremely reliable, extremely performant, fairly inexpensive, multi-region (global). No maintenance, no scaling, almost no tunables, it just works.
Google provides a Pub/Sub emulator for local development.
I don't really buy the vendor lock-in thing for Pub/Sub-like systems. The Cloud Pub/Sub usage pattern is basically the same as Kafka, you can have a library that abstracts away the differences. There are open source libraries that do that[1]. If you ever need to switch cloud providers, or want a messaging system to span cloud providers, you can switch without changing lots of code.
I don't think it's that simple unless I misunderstand how GCP Pubsub works. I don't think GCP PubSub will give you deterministic delivery order within a partition the way Kafka will: https://cloud.google.com/pubsub/docs/ordering
Are you saying GCP pubsub have bad reliability? I never used it but would be surprised if it was down for several hours each weak or would just drop your data.
One of the teams was complaining about behavior on load. Now another team will migrate that flow to Kafka. Another team was complaining about reliability/speed. I think eventually they stopped relying on that.
Google provides a Pub/Sub emulator for local development.
I don't really buy the vendor lock-in thing for Pub/Sub-like systems. The Cloud Pub/Sub usage pattern is basically the same as Kafka, you can have a library that abstracts away the differences. There are open source libraries that do that[1]. If you ever need to switch cloud providers, or want a messaging system to span cloud providers, you can switch without changing lots of code.
[1] https://github.com/google/go-cloud/tree/master/pubsub