AI Quality Assurance: Answers to Common Questions

AI quality assurance raises the same questions for almost every team that starts down this road: what changes, what breaks and what to learn first. The answers below draw on QAlified’s testing guides, the ISTQB certification pages and the documentation of the tools mentioned.

What is AI quality assurance?

It covers two jobs. One is assuring the quality of systems that contain AI. The other is using AI to make QA itself faster. QAlified’s AI testing page focuses on the first: models that behave non-deterministically can return different answers to the same input, so results stop being a simple pass or fail and call for several metrics. Models can also learn and adapt over time, which adds another layer of complexity to validation. Without structured testing, QAlified says, AI-based systems may show variable responses, data bias and security issues that reach users before anyone notices.

Why does traditional QA struggle with AI-based software?

Traditional QA assumes the same input gives the same output. QAlified lists three assumptions that AI breaks: identical inputs can give different outputs, behavior follows a model instead of explicit rules, and retraining can change behavior without a single code change.

The failure modes change too. Instead of bugs and regressions, teams face drift, hallucinations and bias. A suite that is entirely green can still ship an assistant that invents facts.

How do you test a language model?

Anyone new to the topic can start with this AI testing overview, then return here for specifics. QAlified’s LLM testing guide describes the toolkit in three parts:

• Prompt libraries that simulate real user inputs.
• Evaluation functions that score generated responses.
• Automated feedback loops that detect regressions.

Checks look at hallucinations, accuracy, fairness and fit with business requirements. For retrieval systems, the guide adds faithfulness and citation accuracy, and it mentions using a second model as a judge when rule-based checks are not enough.

The guide also separates evaluation from testing. Evaluation measures a model’s overall capability on benchmarks and helps with model selection. Testing checks that the model behaves correctly in your specific application, for instance whether a support bot avoids giving financial advice. Mature teams use both: one checks potential, the other checks readiness.

Which metrics matter most?

It depends on the use case, but QAlified’s metrics guide names six that come up again and again:

• Correctness against verified sources.
• Answer relevancy to the user’s question.
• Task completion, including format and business rules.
• Hallucination rate, meaning made-up or unverified content.
• Contextual relevance for retrieval systems.
• Bias and toxicity.

The same guide warns against leaning on BLEU and ROUGE alone, since they do not measure semantic accuracy or user intent. It also notes that a high hallucination rate in AI-assisted test automation can lead to invalid locators, fake APIs or incorrect assertions, which hurts test reliability directly.

Which standards and training exist?

The ISTQB Certified Tester AI Testing syllabus, version 2.0, teaches the AI-specific quality characteristics defined by ISO/IEC 25059. The exam has 40 questions and a 60 minute limit, and a passing score is 29 points. The Foundation Level certificate is a prerequisite. The audience includes testers, test managers, data analysts, developers who build AI-based systems and user acceptance testers.

The syllabus follows the lifecycle: input data testing, model testing and machine learning development testing, plus a chapter on testing generative AI and large language models. Anyone mainly interested in using generative AI to support testing activities should look at the separate CT-GenAI certification instead.

Version 1.0 is being retired, so plan study time accordingly: the English version stays available until April 21, 2027, and other languages until October 21, 2027.

Can AI help run the tests too?

Yes. QAlified says AI optimizes repetitive QA tasks and improves fault detection, which speeds up validation and lowers operating costs. In practice, Playwright’s agents plan scenarios, generate test files and repair failing tests, while visual tools such as Applitools Eyes check layouts and WCAG color contrast.

The catch is oversight. Someone still has to read what the agents produce and decide which failures are real.

Playwright’s healer, for example, suggests patches such as a locator update or a wait adjustment, and it skips a test when it believes the functionality is broken. That skip is a signal for a human, not a pass.

What happens after release?

Quality work continues. QAlified warns that AI systems can drift after launch and that without monitoring, quality degrades silently, so it recommends monitoring in production and embedding checks into MLOps pipelines.

Langfuse, an open-source AI engineering platform, offers tracing for LLM calls and evaluation features that measure output quality and monitor production health. Whatever you choose, decide who reads the dashboard and what triggers a rollback.

QAlified also suggests capturing performance baselines, visualizing trends on dashboards and setting alerts and thresholds that flag when performance degrades or behavior turns abnormal.