White paper · AI in Business · 17 pages
Assuring AI quality takes more than testing
From QA to EvalOps: how we document that an AI system solves the right task at the required quality, including after it goes into production.
An AI solution is not good because it works in a demo. It is good when we can document that it solves the right task at the required quality, including after it goes into production.
A classic software test has a single correct answer: given a specific input, we expect a specific output. Generative AI has no single correct answer. Two different responses can both be good, and a response can sound convincing and still be wrong. A green test suite therefore does not tell you whether the AI system solves the task.
EvalOps makes quality measurable and repeatable. Success is defined from the actual workflow and tested on a dataset that resembles reality. Evaluation runs automatically on every change, blocks a deployment on regression and continues after the system is in production.
Written for
For those who build or are responsible for an AI system that is in production or about to go into production: AI teams working with RAG solutions and agents, and the people who own the workflow the system supports. It is particularly relevant for organisations in regulated casework, where quality must be documented to supervisory authorities and appeals bodies.
Contents
- 1.Why classic QA falls shortpage 3
- 2.From gut feeling to measurable qualitypage 4
- 3.Start by defining what success meanspage 5
- 4.Build an evaluation dataset that resembles realitypage 6
- 5.Measure both the end result and the componentspage 7
- 6.Automate evaluation, and calibrate it against humanspage 9
- 7.Make evals part of the development processpage 11
- 8.Evaluation does not stop at go-livepage 12
- 9.The most important eval may lie outside the AI systempage 13
- 10.EvalOps and the AI Actpage 14
- 11.EvalOps does not replace security and classic QApage 15
- 12.From AI project to production disciplinepage 15
