Thanks for sharing your blogpost. We had a similar journey. I installed and tried both Langfuse and Phoenix and ended up choosing Langfuse due to some versioning conflicts on the python dependency. I’m curious if your thoughts change after V3? I also liked that it only depended on Postgres but the scalable version requires other dependencies.
The thing I liked about Phoenix is that it uses OpenTelemetry. In the end we’re building our Agents SDK in a way that the observability platform can be swapped (https://github.com/zetaalphavector/platform/tree/master/agen...) and the abstraction is OpenTelemetry-inspired.
As you mentioned, this was a significant trade-off. We faced two choices:
(1) Stick with a single Docker container and Postgres. This option is simple to self-host, operate, and iterate on, but it suffers from poor performance at scale, especially for analytical queries that become crucial as the project grows. Additionally, as more features emerged, we needed a queue and benefited from caching and asynchronous processing, which required splitting into a second container and adding Redis. These features would have been blocked when going for this setup.
(2) Switch to a scalable setup with a robust infrastructure that enables us to develop features that interest the majority of our community. We have chosen this path and prioritized templates and Helm charts to simplify self-hosting. Please let us know if you have any questions or feedback as we transition to v3. We aim to make this process as easy as possible.
Regarding OTel, we are considering adding a collector to Langfuse as the OTel semantics are currently developing well. The needs of the Langfuse community are evolving rapidly, and starting with our own instrumentation has allowed us to move quickly while the semantic conventions were not developed. We are tracking this here and would greatly appreciate your feedback, upvotes, or any comments you have on this thread: https://github.com/orgs/langfuse/discussions/2509
So we are still on V2.7 - works pretty good for us. Havent tried V3 yet, and not looking to upgrade. I think the next big feature set we are looking for is a prompt evaluation system.
But we are coming around to the view that it is a big enough problem to have dedicated saas, rather than piggy back on observability saas. At NonBioS, we have very complex requirements - so we might just end up building it up from the ground up.
"Langsmith appeared popular, but we had encountered challenges with Langchain from the same company, finding it overly complex for previous NonBioS tooling. We rewrote our systems to remove dependencies on Langchain and chose not to proceed with Langsmith as it seemed strongly coupled with Langchain."
I've never really used Langchain, but setup Langsmith with my own project quite quickly. It's very similar to setting up Langfuse, activated with a wrapper around the OpenAI library. (Though I haven't looked into the metadata and tracing yet.)
Functionally the two seem very similar. I'm looking at both and am having a hard time figuring out differences.
I'm a maintainer of Opik, an open source LLM evaluation and observability platform. We only launched a few months ago, but we're growing rapidly: https://github.com/comet-ml/opik
This severely limits the usefulness of a product like this.
Core aspects of the product like workflows and task management should not be tied to a chat vendor in my opinion, and would make me extremely nervous as a potential buyer due to your complete dependence on what SF does with Slack.
I’ve also worked places that strongly dislike Slack and won’t touch it since it was acquired by Salesforce. Ironically, your product would cause Shadow IT deployments (of Slack) in such environments.
Sharing these concerns because I think the product is a really useful concept, but your roadmap for these core functions would mean the difference between considering and completely passing over AccessOwl, i.e. for some subset of potential customers, the hard dependency on Slack is a complete blocker.
Depending on the point of view it can also be a strength. Actually many of our customers like that we're in Slack because their people are already there:
- no login required to request an access
- they don't need to "learn a new application"
So for end users that's great. There is still a web app for admins with more details.
But I can see where you're coming from. We plan to offer an alternative to Slack to be independent if the customers want that.
It sounds like you may have found your niche with existing Slack customers and if that works for you that’s great.
I don’t agree that this is a “strength”, because it limits the growth potential of the product while coupling critical functions to the whims of a 3rd party vendor. I absolutely do see how it’s beneficial for you in this early stage because it allows you to deliver a straight-forward experience for this particular user base (Slack customers) without building your own UIs. But that position of strength is fundamentally limited to that specific group. Move beyond it, and not only would using the product now require the adoption of a non-standard chat tool, but the core function of your product is completely orthogonal to chat making the Slack requirement also appear really odd. That group won’t have muscle memory for Slack or know all of its key features. That group will not benefit from any of the familiarity your current customers find compelling.
And back when I was a Slack customer (I actually like Slack and prefer it to the alternatives) I’d still be raising concerns because of the tight coupling with Slack features.
Not trying to just criticize your decisions here, but trying to elaborate on an outsider’s perspective as someone who has been in the position to bring this kind of product on board at large companies, and as someone who has dealt with the pitfalls of building products that have 3rd party integrations.
Best of luck to you on all of this and it’s good to hear there’s an alternative on the roadmap.
From a legal point of view that might be true, but I believe people are not aware that this is a problem. They just register, check the "Agree terms of service" box and do whatever they want to do. I saw that often, especially with Marketing.
It's not just legal but also the practical point of view. They committed fraud when they clicked that checkbox. It's exactly the same as signing a contract with someone else's name.
Not really, most (larger) companies have internal policies about that, listing the acceptable licenses. Which is exactly what I said - the employees are given the power to accept the terms. Some employees can sign SaaS contracts, though that's usually much less people.
Which makes it even worse because you cannot detect that then :/
Shouldn't people just be able to try out new things? How can a company be innovative otherwise?
And at a specific point (e.g. putting customer data into it), they need to start a proper vendor assessment process.
People can absolutely try new things, but time and time again you cannot trust people to not put sensitive data into those platforms and they continually do.
It's always a balance of information security awareness, culture and technological solutions within an organisation.
Hey Mathias! Was fun chatting about Livebook the other day and yes, I'm definitely looking at it for inspiration! Alas it's an Elixir only notebook and so far as I know, there's very few data folks using Elixir so might be a hard sell.
