<p>The talk begins with a company where generative AI (GenAI) is already in production, but nothing is tested. Every change is risky, and nobody can tell whether the system has improved or simply broken something.</p> <p>Marcin Szymaniuk explains how to bring structure from three perspectives:</p> <ul> <li>Developer: how to build evaluation loops, datasets, and basic tests for retrieval-augmented generation (RAG), retrieval, and document pipelines</li> <li>Team lead/manager: what is realistic to test, what is expensive, and how to plan work around uncertainty</li> <li>Business: how to move fast without guessing and how real user feedback is more valuable than any single metric</li> </ul> <p>He then covers how to evaluate non-deterministic systems:</p> <ul> <li>RAG and document-quality metrics, including relevance, faithfulness, and completeness</li> <li>Conversation testing using synthetic scenarios</li> <li>Using large language models (LLMs) as judges when rule-based checks fail</li> </ul> <p>Marcin also discusses the trade-offs: not everything should be automated, and not everything needs perfect scoring. Testing GenAI is not about achieving perfect correctness - it is about building enough confidence to ship, iterate, and avoid breaking what already works.</p>