<p>AI prototypes are easy to demo and surprisingly hard to trust. A copilot gives a convincing answer, an AI agent completes a workflow once, and a prompt looks better after a few tweaks - but how can a team know whether the system actually improved? How can it catch regressions, measure quality, control cost, and decide whether a new version is ready for production?</p> <p>In this practical session, Eugene Sergueev shows a step-by-step evaluation playbook for large language model (LLM) applications and AI agents. The talk focuses on moving from demo-driven AI development to a repeatable engineering process with real-world cases, quality rubrics, baselines, evaluation runs, thresholds, release gates, and production feedback loops.</p> <p>The session walks through a lightweight reference architecture that teams can adopt without building a large evaluation platform first. Eugene explains how to collect 20–50 representative real-world cases, define what “good” and “bad” mean, create a golden dataset, compare prompt, model, and agent versions, and detect regressions before they reach users.</p> <p>The talk also covers where rule-based checks, human review, and LLM-as-a-judge approaches work well - and where they fail. Eugene discusses how teams can use shadow runs, A/B tests, expert corrections, and production feedback to improve AI systems in small, measurable steps.</p> <p>Attendees will leave with a practical framework they can reuse in their own teams: an evaluation loop, a quality rubric structure, a release-readiness checklist, and a reference architecture for evaluating LLM-powered workflows and AI agents.</p> <p>The session is designed for engineers, AI practitioners, product engineering leaders, and platform teams who have already seen impressive AI demos and now need a way to make AI systems reliable, measurable, and safe enough for real production use.</p> <p>This is not a vendor or platform demo. The session focuses on practical engineering patterns that can be applied across different AI stacks: internal copilots, LLM-powered product features, document question-answering (QA) systems, AI-assisted development workflows, and tool-using agents.</p> <p>The talk is structured as a step-by-step implementation guide and includes:</p> <ul> <li>a lightweight evaluation loop for LLM apps and agents;</li> <li>an example structure for a golden dataset;</li> <li>a quality rubric for AI outputs;</li> <li>examples of metrics and thresholds;</li> <li>a release-readiness checklist;</li> <li>common anti-patterns when teams evaluate AI manually;</li> <li>guidance on where human review, automated checks, and LLM-based graders should be used.</li> </ul>