CEO @ traceloop. Ex-Google, formerly chief architect at Fiverr. Over 15 years of experience building software. M.Sc in CS from the Hebrew University in Jerusalem.
@
nir_ga
September 2025
Hallucinations in RAG apps undermine trust when answers stray from retrieved context. This post explains how to detect them automatically using Faithfulness checks, an LLM-as-a-Judge, and claim-by-claim verification.
Read more →September 2025
Manual spreadsheet checks don’t scale for LLM evaluation. This post shows how to build an automated framework with datasets, evaluators, monitoring, and alerts to ensure faithfulness and relevance at scale.
Read more →September 2025
Debugging multi-step LLM agents is complex without proper visibility. This guide shows how to trace agent execution using observability techniques by capturing prompts, tool calls, retrieval steps, and evaluation metrics to pinpoint failures with precision. Learn what signals to log, how to instrument your pipeline with OpenTelemetry, and how to analyze common failure scenarios. Finally, see how platforms like Traceloop streamline tracing, versioning, and monitoring so you can move from guesswork to clear insights.
Read more →September 2025
A/B testing prompts isn’t just about swapping lines of text. It’s a structured process. This guide breaks down the five key steps to running reliable A/B tests for LLM prompts in production, from forming a clear hypothesis and defining robust metrics to splitting traffic safely and analyzing results with statistical rigor. Learn how modern platforms like Traceloop simplify prompt management, evaluation, and deployment so you can continuously improve user satisfaction and performance.
Read more →September 2025
RAG apps can break in production even when the code hasn’t changed. The culprit is rarely a bug. More often, the cause is data drift, concept drift, flawed chunking, embedding limitations, noisy context, or external dependencies. This article explains the hidden causes of silent degradation in RAG systems and how continuous monitoring and evaluation can help keep outputs accurate, relevant, and trustworthy over time.
Read more →September 2025
Learn how to evaluate whether your LLM outputs are truly satisfying users. This guide breaks down a practical 5-step framework to measure user satisfaction, track implicit signals like retries and abandonment, detect hallucinations, collect explicit feedback, trace root causes, and build a continuous improvement loop. Perfect for teams looking to turn unpredictable LLM behavior into reliable, high-performing applications.
Read more →July 2025
Traceloop auto-instruments your LangChain RAG pipeline, exports spans via OpenTelemetry, and ships ready-made Grafana dashboards. Turn on the built-in Faithfulness and QA Relevancy monitors in the Traceloop UI, import the dashboards, and set a simple alert (e.g., > 5 % flagged spans in 5 min) to catch and reduce hallucinations in production, no custom evaluator code required.
Read more →March 2024
Tokenization, the process of breaking text into smaller units called tokens, forms the foundation of how Large Language Models like BERT and GPT understand and generate human language. This article explores different tokenization algorithms, their applications in Natural Language Processing tasks, common challenges, and how they compare to other text evaluation metrics, highlighting tokenization's crucial role in bridging human and machine understanding.
Read more →
April 2024
Deploying Large Language Models (LLMs) requires careful planning across security, infrastructure, and monitoring to ensure successful production implementation. This article provides a practical checklist covering essential steps from defining objectives to gathering user feedback, helping teams navigate the complexities of LLM deployment while maintaining optimal performance.
Read more →
February 2024
While Large Language Models (LLMs) are commonly used to evaluate other LLMs' performance, new research shows they produce highly inconsistent scores when rating the same text multiple times, making them unreliable as evaluation tools. The article suggests using established, deterministic metrics like BLEU and ROUGE instead, or implementing statistical methods that account for this scoring variability, to more accurately compare different LLM models and prompts.
Read more →
October 2023
OpenLLMetry has been released as an open-source extension to OpenTelemetry, providing observability tools for LLM and AI applications without vendor lock-in, similar to how OpenTelemetry revolutionized cloud observability. The tool offers instrumentations for various LLM platforms (like OpenAI and Anthropic), vector databases, and frameworks, allowing developers to track LLM responses, prompts, and performance metrics while maintaining the flexibility to use any observability platform they choose.
Read more →
Start tracking, testing, and improving your LLMs with Traceloop — one line of code, full observability.
