In places I have worked at that used Databricks, I feel they chose it for the same reasons big orgs use Microsoft: it comes out of a box and has a big company behind it. Technical benchmarks or even cost considerations would be a distant second.
Are there really any other managed Spark/full data platform providers that come close to the level of ergonomics and maturity of Databricks? I agree some of their features are half-baked and you can turn on some dangerous cash faucets if you don't have a very diligent administrator, but I have not come across anything that our developers (fresh grads up to seasoned Spark veterans) liked as much as Databricks.
I worked at a company that dumped Databricks once the first bill came. I guess it was an order of magnitude more expensive than what they expected. It was less expensive to rebuild the pipeline from scratch with a different product.
There are real advantages from having a managed data platform compared to managing everything yourself, especially if you have a large number of data teams that need to collaborate.
Yep, and Databricks will have you churning and changing everything on your stack every 18 months (if you want to keep up to date at all) - its not what I would choose as a data partner unless I was just picking what all the other kids at lunch were.
