https://www.horsewithapointyhat.com/tags/llm-as-a-judge/Recent content in LLM-as-a-Judge on Horse with a Pointy HatHugoen-usFri, 05 Jun 2026 00:00:00 +0000https://www.horsewithapointyhat.com/posts/its-llms-all-the-way-down/Fri, 05 Jun 2026 00:00:00 +0000https://www.horsewithapointyhat.com/posts/its-llms-all-the-way-down/<p><strong>This is a reposting of a blog I wrote for the <a href="https://technology.complyadvantage.com/its-llms-all-the-way-down-a-practical-guide-to-genai-evals/">ComplyAdvantage Tech Blog</a></strong></p> <p>Generative AI (GenAI) systems and Large Language Models (LLMs) are empowering us to tackle new types of problems and enabling the implementation of smart, autonomous (or semi-autonomous) systems. However, the history of responsible machine learning and data science is rooted in the need to quantify and monitor the performance of the models we use.</p> <p>In the brave new world of GenAI, new challenges arise due to more complex modes that are inherently non-deterministic and for which evaluation is much more nuanced, given the nature of the outputs. In this scenario, we cannot purely rely on classic numerical metrics such as Recall and Precision, often derived from exact matching of strings. GenAI solutions can range from simple single-shot calls to an LLM to complex Agentic AI workflows that incorporate Retrieval-Augmented Generation (RAG), deterministic tools, and sub-agents; each component needs its own form of evaluation in addition to an end-to-end performance measurement.</p>