Quality Gates for ML: 4 Layers Between Training and Production

40% of our candidate models got rejected at the quality gate. That is not a failure rate. That is a protection rate.

Without quality gates, every model that finishes training goes to production. Good models. Bad models. Models trained on corrupted data. Models that score well on the test set but tank in production.

Quality gates ask one question before every deployment: is this model actually better than what we have?

Four Layers of Quality Gates

Each layer catches what the previous one missed. Defense in depth for ML.

The Fixed Test Set Rule

Every comparison uses the same test set (random_state=42). If you let the test set change between evaluations, you are comparing data, not models.

Different test sets between champion and candidate? You are not comparing models. You are comparing data.

The fixed test set is the scientific control. Without it, your quality gate is theater.

Layer 1: Registration Gate

A model that does not pass minimum thresholds never enters the MLflow Model Registry:

1
2
3
4
5

MIN_F1 = 0.20
MIN_NET_BENEFIT = 40000

if f1_score < MIN_F1 or net_benefit < MIN_NET_BENEFIT:
    return "NOT_PROMOTED"

The registry stays clean. No experimental garbage polluting production aliases. Only models worth comparing ever get registered.

Layer 2: Champion vs Challenger

The candidate must be strictly better than the current champion. Ties go to the champion (less complexity, already validated).

1
2
3
4
5
6
7
8

champion = mlflow.sklearn.load_model(f"models:/{MODEL}@champion")
candidate = mlflow.sklearn.load_model(f"models:/{MODEL}@candidate")

champ_score = calculate_net_benefit(y_test, champion.predict(X_test))
cand_score = calculate_net_benefit(y_test, candidate.predict(X_test))

if cand_score > champ_score:
    client.set_registered_model_alias(MODEL, "champion", candidate_version)

Same test set. Business metric. Strict comparison. (See Part 16: ML Governance for the alias pattern.)

Layer 3: The CI/CD Pipeline Gate

The single most powerful gate. GitHub Actions runs the full pipeline. Job 2 trains and compares. If the candidate loses, Jobs 3-6 are skipped:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

jobs:
  train-and-gate:
    outputs:
      gate_result: ${{ steps.gate.outputs.result }}

  build-container:
    needs: train-and-gate
    if: needs.train-and-gate.outputs.gate_result == 'PROMOTED'

  deploy-gitops:
    needs: [export-model, build-container]
    if: needs.train-and-gate.outputs.gate_result == 'PROMOTED'

When the candidate loses:

• Workflow status: SUCCESS (green checkmark)
• Jobs 3-6: SKIPPED
• Production: UNCHANGED
• Champion: STILL SERVING

The pipeline did not fail. The pipeline protected production. A skipped deploy is a successful quality gate.

Layer 4: Canary Gate

Model passed all three gates. Deploy via canary with KServe (80/20 split). Monitor errors, latency, prediction distribution. If anything breaks, automatic rollback.

The previous layers compare on a test set. The canary layer tests against real production traffic. That is the final truth.

The DevOps Parallel

You already have quality gates. You just call them something else.

Same mechanism. Different metric. Net benefit instead of test coverage.

Five Anti-Patterns That Break Quality Gates

When the Gate Fails: What To Do

A rejected model is the system working. Then:

1. Diagnose the comparison. Champion $45K vs Candidate $38K? Why did net benefit drop?
2. Check the data. dvc diff for training data changes. Class imbalance shift?
3. Check the model. Confusion matrix side-by-side. Threshold sensitivity shifted?
4. Decide. Bad data? Fix the pipeline. Genuinely worse? Champion stays.

The gate is not the enemy. Bad models in production are the enemy.

Quick Reference

This is Part 19 of the MLOps for DevOps Engineers series. Hands-on MLOps courses are available at stacksimplify.com/courses. For weekly updates, join the newsletter.