SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-model-ab-test

What does an online A/B test compare for two model versions?

A baseline control and a variant on split live traffic.

Start playing free

  1. Two models on unrelated populations
  2. Control and variant under one experiment ✔
  3. Training loss from different years

Why: Users or requests are assigned to experiment arms, and outcome metrics are compared. Traffic checks and confidence intervals help distinguish an effect from setup bias or random variation.

Huh, didn't know that: A statistically significant lift can still be a bad launch if safety or latency worsens.

The dad joke
Model A and Model B split the traffic because neither liked carpooling.
Like this card?
Play the game

References

Answer: Control and variant under one experiment. Users or requests are assigned to experiment arms, and outcome metrics are compared. Traffic checks and confidence intervals help distinguish an effect from setup bias or random variation.

  1. Google Cloud A/B Experiment Monitoring docs.cloud.google.com/retail/docs/a-b-monitor
    Each experiment has a Monitoring tab displaying traffic split metrics that can help you determine if the A/B test is set up correctly.
    Checked 2026-10-10.
  2. Google Cloud MLOps Guide docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
    production deployment of a new version of an ML model usually goes through A/B testing or online experiments
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhy document a dataset's intended use and context?Artificial IntelligenceWhy can inference consume more total energy than one training run?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?

That's stuff everyone should know.