How to Test AI Features Before Your Users Do

LLM-powered features often succeed in demos and still fail in production. A prompt change, a model upgrade, or a change to the tools an agent can call can quietly reduce answer quality, break grounding, or introduce risky behavior.

This talk shows engineers and tech leads how to build a practical eval loop for AI features using the same mindset they already bring to testing software. Using an anonymized internal agent that turns technical incident updates into executive summaries as the running example, we will start with a small app that appears to work, then expose three realistic failure modes: unsupported claims, missed key information, and brittle behavior after a change. From there, we will build a lightweight eval suite that turns those failures into repeatable checks. The demo will cover a compact golden dataset, simple scoring patterns, and a regression workflow for prompt, model, and tool changes.

Attendees will leave able to explain the difference between benchmarks and app evals, compare pass/fail, rubric, and pairwise scoring, implement a small but useful eval dataset, and troubleshoot whether failures come from tool calls, generation, or overall system design.

Improving is providing pizza at 6:00 and the presentation will begin at 6:30.

RSVP - Attend Event

Speaker Bio

Ron Dagdag has spent 25+ years figuring out which tool fits the problem—and when “good enough” is the right engineering choice. Today, he leads AI research engineering at Thomson Reuters, building agentic systems that hold up in production, not just on stage.

A Microsoft MVP since 2017, Ron is currently recognized in Web Development and Microsoft AI Foundry, with previous awards in Windows, Mixed Reality, and IoT. He speaks internationally, organizes community events across DFW, and keeps tinkering with agents, robots, and half-finished ideas.

He builds things, tests what breaks, and teaches what works.