Release Notes
July 2024
New Metrics
Continuing the tradition from last month, here are some more new metrics we introduced to help you understand the performance of your LLM:
- QA Relevancy: Asses the relevant of an answer generated by a model with respect to a question. This is especially useful when running RAG pipelines.
- Faithfulness: Checks whether some generated content was inferred or deducted from a given context. Relevant for RAG pipelines, entity extraction, summarization, and many other text-related tasks.
You can set set any of these metrics in our Monitoring dashboard, to automatically detect issues in every workflow execution in your production system.
Demo Time
Wondering how those new metrics actually work in production? Check out the demo we built. We used LlamaIndex and Streamlit to index our entire documentation, and built a chat app that allows you to ask any question in the world (as long as it's about Traceloop or OpenLLMetry). Check it out, try to make the model hallucinate and see how Traceloop catches the errors in real-time.
New Integrations
- We partnered with Posthog, so you can now view all the data from Traceloop right in your Posthog instance. See the full docs for more information, and let us know if there's anything you're missing from this integration.
- We're also partnering with OpenPipe so that you can use the data from Traceloop to fine tune your models! Ping us if you want early access. We expect to have this available to everyone later this Q.
In the News
New Relic partners with us to offer observability with OpenLLMetry.
Cerebral Valley published a deep-dive about Traceloop + an interview with me.
Elixir is a cool new startup that offer voice AI observability with OpenLLMetry.
