Last year on these pages we covered a range of topics from ML testing basics to formal verification and drift. This year we’ll be picking up those threads again and more. AI / ML deployment has continued apace and it is hard to keep up with new developments! What is clear is that the need for good validation, assurance and robustness has only grown:

• Autonomous flight plans are advancing in many parts of the world. Zipline has announced autonomous drone delivery for Huston and Phoenix in the United States. Walmart and Wing announced a major expansion of drone delivery to 100 additional U.S. stores, enabled by operational authorizations from the Federal Aviation Administration. In other places like the United Arab Emirates, authorities began mapping air corridors for air taxis and cargo drones; in parallel, there are plans for public air-taxi operations in Dubai in 2026.

• Driverless cars are set to reach more cities with Waymo coming to London in 2026.

• Enterprise deployments of AI based Agents is on the agenda for almost every large company, many of which are designed to work fully autonomously.

• In many cases organizations are no longer just building their AI systems and agents from scratch but using off-the-shelf frameworks and components. This adds an extra layer of challenge to the task of validation since teams often don’t have full access to the model itself.

This year we’ll dig further into testing and validation of AI to meet these challenges. For now though, here are five of our favorite posts from last year!

• ML Testing Refresher - Aka Skyscrapers and Rocks (https://resilient.safeintelligence.ai/p/ml-testing-refresher-ae9). This article covers why testing and validation of ML is so hard (and necessary).

• The ML Benchmarking Primer (https://resilient.safeintelligence.ai/p/ml-benchmarking-primer). Going beyond testing to benchmarking and continual performance improvements.

• Formal Verification of ML (https://resilient.safeintelligence.ai/p/formal-verification-of-ml). What happens when you want to go beyond testing and deeply analyze the properties of a model.

• VNN-COMP: Benchmarking the Verification of Neural Networks (https://resilient.safeintelligence.ai/p/vnn-comp-benchmarking-the-verification). Covering the trailblazing competition that helps push the boundaries of what’s possible in formal verification each year.

• When Machine Learning Models Stop Seeing Clearly (https://resilient.safeintelligence.ai/p/when-machine-learning-models-stop). A nice primer on what drift is and why it needs to be kept in check!

We’re looking forward to a great year in safe AI adoption!

A vision model trained on London’s summer mornings can start losing confidence by November. The city turns reflective; to the human eye, the scene is still London, but to the model, it’s an entirely new world. This is drift, not a server crash, not an obvious bug, but a quiet decay that begins the moment the world changes faster than the model can keep up.

Machine learning systems rarely fail overnight. Their decline is gradual, almost imperceptible. The model keeps running and predictions keep flowing, yet beneath the surface, a silent divergence grows between what the model remembers and what it sees. Over time, what began as a few misclassifications on a foggy morning can grow into systemic misjudgement, not because the model failed, but because London moved on, and the model didn’t.

What exactly is Drift?

At its core, drift is the degradation of model performance caused by a divergence between the production and training environments.

It can emerge from anywhere — new camera sensors, shifting seasonal light, or evolving user behaviour. Every model begins life as a frozen snapshot of the world; drift begins the moment the world moves on.

Formally, drift describes the divergence between the joint data distributions seen during training and those encountered in production:

This joint distribution can be decomposed into two components:

So drift may occur in either the input distribution P(X) or the conditional relationship P(y∣X), or both. This gives rise to the main categories of drift observed in machine learning systems.

Types of Drift

Covariate Drift (Data Drift)

The input distribution P(X) changes, while the relationship P(y∣X) remains constant. Imagine a CCTV model trained on bright summer mornings in Oxford Street. By winter, the same scene has now changed. The objects haven’t changed, but the way they appear has.

A time-dependent version, known as temporal drift, happens when input data varies by hour, season, or event. Morning traffic in London looks nothing like midnight traffic. Unless the model encodes these rhythms, performance will fluctuate with time.

Prior Probability Drift (Label Drift)

The overall class or label distribution P(y) changes, even if the inputs look similar. In summer, London’s roads are full of cyclists (y1) and people (y2). Your model is trained on this “normal” distribution P_train(y). In winter, there are far fewer cyclists but a surge in delivery vans (y3). The model’s idea of a “normal” scene is now wrong. It may become over-confident in predicting cyclists (its prior) and less accurate at identifying the newly common vans. This shift in balance is label drift.

This is a critical factor for models sensitive to class imbalance.

Concept Drift

The mapping between inputs and outputs P(y∣X) itself changes. The same input now implies a different meaning. A content moderation model is trained to X (a “thumbs up” gesture) as y1 (”benign”). The company expands to new regions where this gesture is offensive. The human labelling team is now retrained to label the same X as y2 (”offensive”). If the model isn’t retrained, it will fail, a victim of concept drift.

Concept drift can occur in several ways:

• Sudden Drift: An abrupt change in rules or context.

• Gradual Drift: A slow evolution over time.

• Incremental Drift: A sequence of small, cumulative changes.

• Recurring Drift: Old patterns re-emerge cyclically.

Model Obsolescence & Internal Factors

Not all decay comes from data. Some arise within the model itself and are operational, not statistical. This isn’t drift in the strict sense, but staleness as the model stops keeping pace with a changing world. The clearest form is Algorithmic Obsolescence, where architectures or representations age out. A CNN, once state-of-the-art, may falter against newer vision transformers, not because the data changed, but because the model didn’t evolve. As data evolves, weights and hyperparameters tuned for past distributions gradually reflect drifts. This form of decay highlights that a model is an artefact of its time.

This is why continuous retraining pipelines and benchmark refresh cycles are critical MLOps practices, as they provide a mechanism not only to update data but also to upgrade the model’s architecture and training methods periodically

Drift in Computer Vision Systems

In computer vision, drift isn’t abstract, but it’s visible. A model’s perception of the world depends on light, texture, and hardware, so even subtle changes can disrupt its understanding. Data drift occurs when the pixels themselves change: new camera sensors or image processors alter colour balance and noise; compression or resizing in production reduces image fidelity; and seasonal light alters how edges and shadows are captured. The world looks the same to us but differently to the model.

Concept drift is more deceptive. The pixels stay the same, but their meaning shifts. A product classifier for “shoes” may fail when confronted with new designs it was never trained on. A defect detector once accurate for “scratches” and “dents” may struggle with a new category, such as “micro-fractures.” In other cases, the ground truth itself evolves; for instance, content moderation models must constantly adapt as community standards redefine what is considered unsafe.

The MLOps Playbook on How to Monitor Drift

You can’t detect change without a baseline, and monitoring is not a passive activity but an active statistical process. rift detection begins with a reference. A baseline profile captures the statistical fingerprint of your training data with histograms, quantiles, mean, and variance for numeric features; category counts for discrete ones. For vision systems, this is the most critical step. Drift cannot be tracked at the pixel level; instead, you monitor:

1. Proxy Metrics: Distributions of low-level features like brightness, contrast, and sharpness.

2. Embedding Distributions: The statistical profile (e.g., mean, covariance, or a learned density model) of embeddings extracted from a frozen backbone (e.g., ResNet or ViT).

This baseline becomes the “ground truth” against which all future data is compared. With a baseline established, data drift monitoring uses statistical tests to assess how new data deviates from the baseline.

Statistical Distance Methods (for Low-Dimensional Proxies)

For 1D proxy metrics (like brightness) or categorical features, classical statistical tests are highly effective and explainable.

High-Dimensional and Streaming Methods (for Embeddings)

Classical tests lose power in high-dimensional embedding spaces. For this, more advanced methods are required.

Monitoring Concept Drift

Concept drift is the hardest to monitor because it concerns meaning, not just data. Detecting it requires ground-truth labels, which in production are often delayed, sparse, or expensive. When labels aren’t immediately available, monitoring must rely on indirect signals.

Without Labels (Indirect Monitoring)

In the absence of new labels, the most practical approach is to observe the model’s own behaviour. Track the distribution of predictions over time: if a model that once predicted [80% A, 20% B] now outputs [50% A, 50% B], its internal representation of the problem has shifted. Similarly, monitor confidence entropy; a rise in uncertainty or wider spread of confidence scores often precedes measurable accuracy decay. These indicators don’t prove drift, but they are early warnings that the relationship between inputs and outputs may have changed.

With Labels (Direct Monitoring)

When ground truth becomes available, drift detection becomes concrete. Track performance metric over rolling windows to reveal gradual degradation. To separate real decay from random noise, apply Statistical Process Control (SPC) techniques, such as EWMA or CUSUM charts, which flag statistically significant performance drops. The most reliable systems maintain a human-in-the-loop, continuously labelling a small, rotating sample of production data (typically 1–2 %). This rolling ground truth acts as a living benchmark, enabling early detection and timely retraining.

Key Takeaways

Building a high-performance model is not the finish line, but a starting point. The real challenge in production machine learning is keeping models relevant and resilient to the inevitability of change. Here are the key takeaways for building a drift-robust MLOps practice:

• Monitoring Is Not Optional: The world will change, and your data will change with it. Continuous monitoring isn’t an add-on but a core requirement for any deployed model. Use a defence-in-depth strategy that includes statistical tests, embedding-level methods, and performance metrics.

• Detection Without Response Is Just Noise: Every alert must lead to action. Adaptation strategies fall into two complementary paths:

  • Data-Centric Mitigation: Fix the data. Use data augmentation to simulate new conditions (e.g., fog, lighting, sensor noise) or active learning to re-label uncertain or drifted samples.

  • Model-Centric Mitigation: Fix the model. Choose full retraining for maximum reliability, incremental fine-tuning for speed, or online learning for continuous, real-time adaptation.

• Close the Loop: Drift is an engineering problem; monitoring pipelines must be automated to trigger alerts, and those alerts must connect directly to governance and retraining workflows.

• Keep Recalibrating: Treat your model like a camera that must be constantly refocused and recalibrated to keep seeing the world clearly. The goal is not a perfect model, but a resilient one.

Further Reading

→ Neptune AI: A Comprehensive Guide on How to Monitor Your Models in Production

→ MIT CSAIL: Data-Centric AI vs. Model-Centric AI

→ Practical MLOps by Noah Gift and Alfredo Deza

Leave a comment

For decades, aviation software has been built on the principle of deterministic logic. Give the system input A, and it will always produce output B. This predictability is the bedrock of aviation safety certification, where every possible path can be tested, verified, and traced back to a requirement. Artificial intelligence breaks that contract. Modern neural networks have shown remarkable skill in vision and perception tasks relevant to flight, from obstacle detection to runway tracking, but their performance gains come at a cost, “complexity“. As these models grow deeper and more data-driven, their inner workings become harder to verify, explain, or even fully reproduce.

This creates what regulators now call the “assurance gap.” How can agencies like the Federal Aviation Administration (FAA) or the European Union Aviation Safety Agency (EASA) certify a system whose decision-making can’t be perfectly predicted or explained?

This is the exact problem that the Concepts of Design Assurance for Neural Networks (CoDANN) framework, a foundational project by (EASA) and Daedalean were designed to solve. Instead of trying to verify the final, opaque neural network, the framework’s breakthrough was to shift the focus to certifying the entire process that created it. This introduced a new methodology called Learning Assurance.

To appreciate this shift, we first need to look at the established standard for aviation software: the traditional V-shaped development cycle.

This model is the perfect picture of deterministic engineering that moves downwards through requirements, design, and implementation and then climbs back up through testing and validation. This framework gives aviation software provable traceability where every requirement is explicitly validated by a corresponding test, and the respective implementation traces back to design decisions.

“The V-model’s logic breaks down where the ‘Implementation’ phase is no longer a human writing explicit, verifiable rules, but a learning system (neural network) that learns implicit patterns from data, creating a system whose logic is not fixed and can evolve as the data itself shifts over time. This phase now consists of the dataset and the training algorithm, two elements that the V-model was never designed to handle. To handle that, CoDANN didn’t discard the V model but rather expanded it into a W model.

Here’s what each stage means in the W-model:

Evolution of CoDANN

The CoDANN (2020) and its W-model provided the foundational strategy (certify the process, not just the product). Its 2021 successor, CoDANN II, immediately extended this framework to practical deployment, tackling the challenges of implementation and inference assurance (like quantisation, hardware mapping, and timing analysis) while also formally adding Explainability, system integration, and robust runtime (Out-of-Distribution) monitoring as core pillars of a trustworthy AI.

Europe’s Path via the EASA formally adopted these concepts in its AI Roadmap 2.0 (2023), making Learning Assurance a core pillar of its official strategy. The goal is pragmatic by using existing safety frameworks, but add the new ML-specific processes and evidence required to scale from simple assistance to higher autonomy.

The US Path via the FAA Roadmap for AI Safety Assurance (2024) followed suit. It sets out the principles for adopting AI within the existing certification scaffolding, emphasising incremental approvals, human-in-the-loop operation, and lifecycle assurance that covers both design-time and operation-time.

To bridge national frameworks, SAE G-34 and EUROCAE WG-114 are developing the first globally harmonised AI assurance rulebook: AS 6983 / ED 324 (Development and Assurance guidelines for Aeronautical Systems and Equipment Implemented with Machine Learning). The forthcoming standard transforms the W-model into an actionable compliance checklist, specifying how to produce certifiable evidence for data management, training, model verification, configuration control, and runtime assurance.

To illustrate what that looks like in practice, the draft ARP 6983 / ED-324 defines explicit objectives for every phase of the Machine Learning Development Lifecycle (MLDL). Each assurance level has its own set of mandatory objectives and distinct assurance checkpoints.

Beyond the Checklist

The AS 6983 standard provides the compliance checklist, but how do engineers actually prove a complex, probabilistic neural network is safe? This is achieved through a multi-layered defence:

1. Formal Verification: This is where the strategy shifts from exhaustive testing to mathematical proof that the model satisfies specific safety rules. It’s impossible to test for every single thing the AI might see, such as every possible weather condition, every type of runway, and every time of day. Instead of just trying to find failures, engineers now prove their absence and find counterexample, read more about formal verification here →.

2. Overarching Properties (OPs): A concept pioneered by NASA, OPs take a pragmatic approach. Rather than proving everything about the AI, you prove a small set of unbreakable safety boundaries. For example, the aircraft will always maintain a safe distance from another aircraft. These rules provide a rock-solid, provable safety guarantee, no matter what the complex AI model thinks it’s seeing.

3. Runtime Assurance (RTA): This is the system’s active safety net that operates during the flight like a co-pilot that never improvises. It assumes that despite the best design, the AI might still encounter a bizarre edge case. A simple, verified controller (the Checker) runs in parallel with the complex AI (the Doer). The AI can make high-level decisions and every command it sends is screened by the Checker. If a command ever looks unsafe or uncertain, say the AI suggests a turn that’s too sharp or a descent that exceeds its limits, the Checker instantly blocks it and replaces it with a safe, pre-approved manoeuvre. This design means that even if the AI encounters something it was never trained on, the aircraft always stays inside a mathematically defined safety envelope.

These proven properties and assurance methods are then all assembled into a formal safety case (a structured, auditable argument), supported by evidence, that the entire system is acceptably safe.

Leave a comment

Just last month, on the 23rd-24th of September 2025, the atmosphere at the AI Engineer (AIE) Paris Expo was electric. This was the first edition outside the US, hosted at STATION F and organised with Koyeb. The event carried a very pragmatic energy, with engineers focused on building and deploying the next generation of AI agents, tools, and infrastructure. But beyond the demos and keynotes, as I moved between the sessions I attended, a deeper personal theme emerged: the future isn’t just about building more intelligent AI, it’s about building more trustworthy AI.

My Key Takeaways

→ Open Source as the Foundation of Trust

Lélio Renard Lavaud (Mistral) reminded us that adoption at enterprise scale hinges on openness. By sidestepping vendor lock-in and surfacing transparency, open models create the reliability businesses demand. Laurent Sifre (H Company) echoed this, framing open-source “bricks” as the future scaffolding for AI innovation.

→ Security and Safety by Design

Jesús Espino (Ona) and Martin Woodward (GitHub) emphasised that trust cannot exist without security. Isolation, auditability, and reproducibility aren’t optional; they are essential, especially in regulated environments. Safe AI deployment protects both users and businesses, ensuring that agentic workflows remain reliable even under high stakes.

→ Learning from Failure: Context, State, and Infrastructure

Thomas Schmidt (Metabase) and Rémi Louf (.txt) reminded us that agents still hallucinate metrics, misreport outputs, and occasionally break fundamental interactions. Emil Eifrem (Neo4j) and talks from Spotify and Shopify highlighted the root cause: poor state and context management. Without clear context and robust infrastructure, agents misstep or stall. Yann Leger (Koyeb) emphasised that resilient, heterogeneous, agent-ready systems are essential to turn these failures into trustworthy performance.

→ Evaluation and Trust in AI Behaviour

If you can’t measure it, you can’t trust it. Pierre Burgy (Strapi), SallyAnn DeLucia (Arize AI), and Srilakshmi Chavali highlighted a hard truth: static benchmarks often fail to reflect real-world reliability. Instead, emerging frameworks focus on session-level monitoring, conversations, and user “vibe” to systematically evaluate AI behaviour. Continuous feedback loops enable agents to learn and improve transparently, turning evaluation into a true measure of trustworthiness.

→ Efficiency, Sustainability, and Scale

From Bertrand Charpentier’s deep dive into compression techniques to Steeve Morin’s sparse attention on CPUs, engineers are proving that trustworthy AI must also be efficient AI. Sustainability isn’t a side goal. It’s part of reliability, cost-effectiveness, and accessibility.

→ Agents Moving from Promise to Practice

The event showed the inflexion point where agents stopped being hype and started showing their promise.

• Codebase rewriting at scale (Spotify)

• Senior-level AI code review (Graphite)

• Agent swarms for refactoring (All Hands AI)

• Multi-agent orchestration (Docker)

Each showed a vision where agents aren’t assistants, they’re teammates.

From the sessions I attended, my strongest impression is that the future of AI won’t be built on hype, but on a foundation of trust. The conference made it clear that trust is the result of a responsible engineering mindset.

Looking Forward

The AI Engineer community is just getting started, as the founders said, and from my own experience, you won’t want to miss what’s next. Mark your calendars and secure your tickets:

• AI Engineer Code Summit | November 20–22, New York, NY

• AI Engineer Europe | April 7–10, London, UK

• AI Engineer World’s Fair 2026 | June 30–July 2, San Francisco, CA

I look forward to seeing you in the next one, know more.

Leave a comment

But there’s still a critical question: who verifies the verifiers?

That’s where the Verification of Neural Networks Competition (VNN-COMP) comes in. Think of VNN-COMP as where benchmarking discipline meets formal verification guarantees. Each year, the competition brings together verification tools from around the world and tests them under the same conditions: identical models, identical specifications, and identical compute budgets. The result is a clear, comparative picture of how today’s verification techniques perform in practice, cutting through theory and marketing claims to see which tools actually deliver.

A short history of VNN-COMP (2020-2025)

From a friendly kickoff to a flagship global competition, VNN-COMP has grown into the central benchmark for neural network verification. The following section covers key milestones per year, and a comparison table provides a high-level overview of the competition from its inception to the present day.

2020

Inaugural "friendly competition" to build community. Self-reported results on non-standard hardware. Three benchmark categories were considered across the ACAS Xu, MNIST, and CIFAR-10 datasets

2021

Rules were formalised, official rankings were introduced, and ONNX, along with VNN-LIB, became the standard model and specification formats. Scoring was strict but straightforward: +10 points for each correctly verified SAT/UNSAT instance, and a –100 penalty for any incorrect result. To ensure fairness, every tool was run on equal-cost AWS hardware, with teams choosing between a CPU-optimised or a GPU-optimised instance. Six benchmark categories were considered across ACAS Xu, MNIST, CIFAR-10 datasets and one benchmark on database indexing.

2022

Fully automated evaluation pipeline on AWS. Each benchmark had a total timeout of between three and six hours, with randomisation of instances being
mandatory this year, with tool tuning per benchmark level. The benchmark cut across six categories. Similar to the previous year, teams could choose from a range of AWS instance types, providing a focus on CPU, GPU, or a mixed combination.

2023

Continued standardised ONNX/VNN-LIB and equal-cost AWS evaluation, while keeping automated pipeline & standardised counter-example format. Required CE “witnesses” (input examples that verify a reported counter-example is real). Broader AWS instance choices, no time bonus and –150 penalty for incorrect result.

2024

Regular and extended tracks and continued ONNX/VNN-LIB format and equal-cost AWS. Fully automated installation and evaluation pipeline and required CE “witnesses” plus no time bonus and –150 penalty for incorrect results.

2025

The official VNN-COMP 2025 report isn’t out yet, but Safe Intelligence was there. Read our takeaways 👇🏼

From the friendly kickoff in 20220 to today's flagship status, VNN-COMP has become the rigorous and automated proving ground for neural network verification itself. It’s where theory meets practice and where trust in AI has to prove itself, because benchmarking alone isn’t enough, and verification without verification is a paradox.

Further Reading

• First Three Years of the International Verification of Neural Networks Competition (VNN-COMP)

• The Fourth International Verification of Neural Networks Competition (VNN-COMP 2023): Summary and Results

• The Fifth International Verification of Neural Networks Competition (VNN-COMP 2024): Summary and Results

• About VNN 2025: 6th International Verification of Neural Networks Competition (VNN-COMP'25)

TL;DR

In our last article, we moved beyond conventional testing and introduced formal verification, a deeper form of validation. Now that your model has cleared every validation gate, the next challenge is to deliver it. This article covers choosing the right inference architectures and walks through progressive rollout strategies that aim to minimise risk in high-stakes industries. Lastly, it distils the industry consensus on scaling, which involves standardising, containerising, and orchestrating to handle operational complexities at scale.

Understanding your deployment needs

Before shipping a model to production, step back and map production requirements to the right serving pattern. Four levers drive this decision:

1. Latency & Throughput: How fast must each prediction return and at what volume? These directly dictate the speed and capacity demands on your system.

2. Data dynamics: When does input data arrive, and how fresh must predictions be?

3. Resource constraints & scalability: Available computing power (CPU/GPU, memory) and the infrastructure's ability to scale up or down with varying loads.

4. Cost & Interaction Model Who/what consumes the predictions, and how price-sensitive is the workload?

Core ML Serving Architectures

With the key criteria in mind, ML models are generally deployed in three main architecture patterns:

1. Online Real-Time Inference (Synchronous)

2. Asynchronous Inference (Near Real-Time)

3. Offline Batch Prediction (Batch Transform)

Each of these has its strengths and ideal use cases. Sometimes different terminology is used (for example, “batch prediction” vs. “offline batch transform” or grouping the first two as both “online” methods, synchronous vs. asynchronous), but the concepts remain consistent. Let’s explore each architecture.

Online Real-Time Inference (Synchronous)

This architecture provides immediate prediction responses with minimal latency. Applications that prioritise speed or interactive responses, like fraud detection at transaction time or cockpit alert systems, find this architecture ideal. Here, the model is served via an API that the users call synchronously (the client sends a request/input and waits immediately for the prediction result). When a request arrives, such as a JSON payload of features, the service loads the model, performs inference, and immediately returns the result. Everything happens within the context of that request, usually in a matter of milliseconds or seconds, as illustrated below:

The system is typically optimised to minimise overhead in response. That might involve keeping models loaded in memory, using fast data stores for features, and even making models lighter or distilling them to run faster. The price of that speed is scalability, such as with more instances or powerful hardware used. Additionally, there is a limit to the throughput achievable with a purely synchronous design; if the request volume exceeds what your servers can handle at the same time, either latency will increase or some requests may fail.

Asynchronous Inference (Near Real-Time)

This architecture addresses the bottleneck of synchronous inference by introducing a message queue between your client and the ML model. The requests are buffered (to absorb bursts), and then stateless workers pull jobs, run inference, and write results to a result store. The client is either alerted or polls a results endpoint to fetch the prediction. Since requests buffer, workers can scale horizontally or even down to zero when idle. The client is either alerted or polls a results endpoint to fetch the prediction. This architecture is illustrated below:

If your latency target is seconds, not sub-100ms, async serving smooths out traffic spikes and converts them into steady, predictable costs while preserving reliability and throughput. Lastly, this architecture provides the avenue to insert preprocessing, enrichment, or post-processing stages directly in the queue flow.

Offline Batch Prediction (Batch Transform)

Offline batch prediction refers to running the model against a dataset slice or entire warehouse on a schedule or trigger and writing the outputs to storage, a data warehouse, or a feature store for downstream use. These batch jobs often run on big data processing frameworks or distributed systems (like Apache Spark, Hadoop, or distributed SQL engines) to churn through huge volumes of input.

By exploiting spot instances and off-peak capacity, you slash costs, and because compute is ephemeral, you never pay for idle GPUs. The catch? Results arrive in minutes or hours, so this is great for back-office analytics but ill-suited to user-facing paths.

Which architecture fits your workload?

Choosing the right inference architecture isn't always straightforward. Use this Q&A-style checklist to guide your decision-making clearly and effectively:

Rollout Strategies: Safely Deploying ML Models in Production

In high-stakes domains like finance and aviation, a “big-bang” model swap (where the old system is shut down and instantly replaced by the new model) is a gamble no one can afford. Modern teams now favour progressive delivery, where new models are rolled out to a small slice of traffic or a safe environment first, validating performance in the wild, then scaling up only when metrics look solid. Let's explore progressive rollout strategies that minimise risk, provide real-world validation, and ensure stability.

Champion/Challenger

In this strategy, you deploy the new model ("challenger") in parallel to the existing production model ("champion"). Both receive the same inputs (mirrored traffic), but only the champion’s predictions directly impact users or operations. The challenger operates in the background, and its outputs are evaluated separately to determine if they outperform the existing model and for other experimental analyses. This is also known as shadow deployment, as the challenger models run in shadow mode (without affecting real outcomes)

Why choose Champion/Challenger?

This approach is excellent when regulations require proof of model efficacy or when the cost of a bad decision is extremely high. With proper performance and governance analysis over time, decisions on when a challenger is ready to take over as the new champion can be made.

Flask-like pseudocode

@app.route("/predict")
def predict():
    features = request.get_json()['features']
    
    # Champion makes the actual prediction
    champion_prediction = model_champion.predict(features)
    
    # Challenger prediction is computed but not returned to users
    challenger_prediction = model_challenger.predict(features)
    
    # Log challengr results separately for later comparison
    log_to_evaluation_db(features, champion_prediction, challenger_prediction)
    
    return champion_prediction

Canary deployment

This deployment strategy incrementally deploys the new model to a small fraction of live traffic. The fraction gradually increases this percentage as confidence builds, continuously monitoring performance and stability.

Why choose canary deployment?

Canary deployments are widely used when you want a cautious, data-driven rollout using real traffic. For example, companies like Tesla and Waymo use such rollouts, as they might enable a new vision model for a small set of cars, then expand if it performs well. The canary approach strikes a balance between innovation and caution, swiftly retracting any underperforming model before it causes widespread harm.

Flask-like pseudocode

CANARY_PERCENTAGE = 0.05  # Start with 5%

@app.route("/predict")
def predict():
    features = request.get_json()['features']
    
    # Route a small fraction of users randomly to the canary (new) model
    if random.random() < CANARY_PERCENTAGE:
        prediction = model_new.predict(features)
        model_version = "canary"
    else:
        prediction = model_current.predict(features)
        model_version = "current"
    
    log_prediction(features, prediction, model_version)
    
    return prediction

A/B Testing

In A/B testing, you split live traffic between your existing model (Model A) and a new model (Model B), assigning each to handle a different subset of users or instances. Both models are statistically compared on key performance indicators (KPIs) to determine which model truly performs better. A/B testing starts with the null hypothesis that both models perform identically. Send live traffic to each, track a key metric, and calculate a p-value for the chance that random noise explains the observed gap. If the p-value is below your threshold (e.g., 0.05), the lift is statistically significant, so safely increase Model B’s traffic or promote it to production. Otherwise, keep Model A (or keep testing).

Why choose A/B testing?

It is primarily utilised in situations where it is ethically and operationally possible to treat different users differently for a brief period. For example, a bank​​ might route a small percentage of transactions through a new fraud model to see if it catches more fraud or flags fewer false positives than the old model.

Flask-like pseudocode

TREATMENT_GROUP = set(load_user_ids_for_treatment())  # e.g., 10% users

@app.route("/predict")
def predict():
    data = request.get_json()
    features = data['features']
    user_id = data['user_id']

    # Determine if user is in treatment (B) or control (A) group
    if user_id in TREATMENT_GROUP:
        prediction = model_B.predict(features)
        variant = "B"
    else:
        prediction = model_A.predict(features)
        variant = "A"


    # Log results for offline statistically analysis
    log_ab_test_results(user_id, features, prediction, variant)
    
    return prediction

Scaling Your Deployment: Serving Hundreds or Thousands of ML Models

Once a single ML model delivers measurable business or safety improvements, rapid scaling becomes inevitable. Soon, you're managing not just a handful but hundreds or thousands of models, each bringing unique operational headaches, such as management complexity, resource contention, and cost efficiency. Leading enterprises, such as Netflix, have addressed these challenges using a proven framework: standardise, containerise, and orchestrate. Explore deeper insights:

• Netflix Techblog: Supporting Diverse ML Systems at Netflix

• DataDog: Scaling ML Serving to Thousands of Models

• Industry Best Practices: Serving Hundreds to Thousands of ML Models

• TDS: Serve hundreds to thousands of ML models – architectures from industry

Actionable Takeaways

1. Pin down your performance requirements, resources, cost and scalability constraints, data dynamics and interactions early; these shape your deployment choices and downstream decisions.

2. Map your use case to the right pattern: real-time for instant results, async for flexible workloads, or batch for large-scale efficiency.

3. Rollout models progressively

• Champion/Challenger: Safely validate a new model alongside the existing one without disruption.

• Canary Deployments: Incrementally introduce new models, catching issues early and safely.

• A/B Testing: Statistically validate model improvements on live traffic before full-scale deployment.

4. Standardise your configurations, containerise your models, and orchestrate your ML deployments. It’ll simplify complexity, reduce operational headaches, and make scaling frictionless.

Table of Contents

• TL;DR

• Understanding your deployment needs

• Core ML Serving Architectures

  • Online Real-Time Inference (Synchronous)

  • Asynchronous Inference (Near Real-Time)

  • Offline Batch Prediction (Batch Transform)

  • Which architecture fits your workload?

• Rollout Strategies: Safely Deploying ML Models in Production

  • Champion/Challenger

  • Canary deployment

  • A/B Testing

• Scaling Your Deployment: Serving Hundreds or Thousands of ML Models

• Actionable Takeaways

Leave a comment

• TL;DR

• What do we want to verify?

• Verification Techniques

  • Constraint-Based Verification

  • Abstraction-Based Verification

• Comparison of Verification Techniques

• The Elephant in the Room: Does it Scale?

• Putting Formal Verification to Work

• Further Reading

TL;DR

In the previous articles, we explored identifying the best-suited model for your use case through testing and benchmarking. But ask yourself: does testing and benchmarking give you the confidence to deploy these models in critical scenarios where a mistake is consequential? Do those 10,000 test cases provide you with everything you need? Perhaps you’re considering the scope and coverage of the testing and benchmarking or the inherent uncertainty that these models will face in their environment. This is precisely where formal verification moves beyond testing. It allows for a deeper analysis of the model itself, systematically searching for the fragilities that lead to failure. By generating concrete counterexamples, it proves where the model breaks, giving you the power to build a truly robust system before it's ever deployed.

Formal verification is not a new concept; it has long been integral to ensuring reliability in hardware and, to a lesser extent, software. According to Wikipedia, “formal verification is the act of proving or disproving the correctness of a system with respect to a certain formal specification or property.” Immediately, four essential components emerge clearly from this definition:

1. System under verification: The specific hardware or software whose behaviour requires to be rigorously checked.

2. Formal specification: Precisely defined requirements or properties the system must satisfy, expressed mathematically rather than informal descriptions.

3. Correctness criterion: Conditions that confirm the system meets the specification or identify cases where it fails.

4. Formal methods: Mathematical techniques and proof engines employed to rigorously prove or disprove whether the system satisfies the given specification.

With these four components in mind, think of formal verification as moving beyond conventional testing by employing rigorous mathematical (formal) methods to provide stronger guarantees, which is particularly crucial when even small mistakes can lead to catastrophic outcomes, like in use cases like autonomous driving, medical diagnostics, security surveillance, malware detection, or high-stakes finance.

Formal verification has a long history in hardware dating back to 1984 with tools like Verilog and now SystemVerilog, tools standardised by IEEE and widely adopted for specifications for designing and verifying hardware. Software adoption lagged, largely due to state‑space explosion (as the number of state variables in the system increases, the size of the system state space grows exponentially). When dealing with software in aircraft or nuclear reactors, it's not enough to merely test a handful of scenarios. You need more guarantees that the software behaves correctly under every conceivable situation. Formal verification isn't merely theoretical but can have a practical, real-world impact. For example, it was successfully employed in the design of Paris’s autonomous Métro Line 14.

But how does this relate specifically to AI systems? The well-known AI researcher Andrej Karpathy highlights software evolution from Software 1.0 (manually written software) to Software 2.0 (software learning from data, ML models) and, more recently, Software 3.0 (AI-driven development).

ML models, the core of Software 2.0, are now central to critical decision-making. However, their reliability is a key concern; while powerful, they are often fragile, failing in response to subtle data shifts that humans would not notice. This brittleness has serious consequences. For instance, minor data variations could lead to flawed loan approvals, while small visual distortions could cause an autonomous system to catastrophically misclassify its environment. Similarly, a critical customer service message could be misinterpreted by an NLP model, leading to significant service failures.

What do we want to verify?

These vulnerabilities (adversarial or otherwise) underscore the critical point that performance metrics and benchmarks alone aren't sufficient. We must rigorously verify specific model behaviours to eliminate these fragilities and blind spots. When failures can cost lives, livelihoods, or vast sums of money, “correctness” must be stated in precise, verifiable terms. The properties most often formalised in such critical systems include, but are not limited to, the following:

• Robustness: Does the model maintain stable outputs despite minor perturbations in inputs?

• Safety: Can we confidently say the model avoids catastrophic decisions under all possible conditions?

• Fairness: Could subtle biases or irrelevant data variations lead to unfair outcomes?

• Consistency: The model must uphold fundamental domain rules, ensuring its outputs never violate basic logical or physical constraints across all valid inputs.

These are the types of behaviours formal verification methods are designed to assess, moving beyond intuition, testing and benchmarking. A quick overview of the verification process is illustrated below:

Verification Techniques

We’ve identified the critical properties, such as robustness, safety, fairness, and consistency, that we want our ML models to exhibit. Now, the next question is, how do we formulate these properties into a ‘verification problem’ that formal methods can solve? This involves translating the desired model behaviours into precise, mathematically checkable statements and then verifying whether the model satisfies these statements universally.

Formally speaking (no pun intended), the verification problem asks:

Given an ML model and a formal specification (a property like safety or robustness expressed in mathematics), does the ML model satisfy the formal specification for all admissible inputs and behaviours?

To solve this verification problem, we generally rely on two main techniques: constraint-based verification and abstraction-based verification.

Constraint-Based Verification

Constraint-based verification converts the verification problem, the input domain, model structure, and property to be checked into a single mathematical system of logical/optimisation constraints, typically fed to a Satisfiability Modulo Theories (SMT) or Mixed-Integer Linear Programming (MILP) engine. The solver treats this system of constraints as a single, complex puzzle. It doesn't check inputs one-by-one; instead, it uses powerful symbolic reasoning and logical deduction to intelligently prune away vast regions of the search space that cannot possibly contain a violation. Its goal is to find one concrete counterexample that satisfies every constraint at once.

Because this search explores all feasible sub-regions, the approach is sound (it never certifies a false property) and complete (it is guaranteed to find a counter-example if one exists). Given enough time and memory, the solver must terminate with either a proof that no violation exists or with a concrete counter-example that satisfies every constraint and therefore constitutes a genuine failure of the property. The price for such exhaustive coverage is the computational cost, which is prohibitively expensive.

Abstraction-Based Verification

Abstraction-based verification trades theoretical completeness for speed and scalability. In other words, it is sound, but it is incomplete (it may stop with “unknown” even though the property is either true or false), especially if it runs out of time or memory. Unlike constraint-based verification, which pushes the exact verification problem into a heavyweight solver, abstraction-based verification takes a lighter, step-by-step route. It watches the input set flow through the model structure (such as layers in a neural net or splits in a decision tree), and after each layer or decision point, wraps all the possible outputs in one slightly oversized “bubble” (which can take different shapes, such as a box, polyhedron, or zonotope). This step-by-step process, also known as bound propagation, produces an over-approximation at every stage, ensuring every real behaviour is safely enclosed within these bubbles. If the final bubble never touches the danger zone, the model is provably safe; if it does touch, the method refines the bubbles or reports “unknown.”

This "bubble" approach has two main advantages. First, it's significantly faster and more scalable than constraint-based verification, because following one bubble at each stage is much simpler than exhaustively checking millions of individual inputs. Second, it provides trustworthy guarantees: if the final bubble sits fully in the safe region, the model is proven safe. However, since each bubble is padded for safety (over-approximated), it can sometimes become too large and overlap the unsafe zone. When this happens, the method can't immediately tell if there's a real problem or just extra padding. At this point, abstraction-based methods usually employ lightweight solvers to quickly determine whether a genuine violation exists within the bubble. If no violation is found, the bubble is refined by splitting the input region further. If time or resources run out before a clear answer is obtained, the method returns "unknown."

Comparison of Verification Techniques

Constraint-based and abstraction-based methods answer the same fundamental question: “Does my model always satisfy the given property?” However, their mathematical approaches differ significantly. To illustrate this clearly, let's first consider a concise comparative overview:

To vividly demonstrate these differences, let's trace how each one tackles a simple verification problem. Consider this tiny neural network and a safety property:

Layer 0 (input): x ∈ [0,1]
Layer 1 (linear): z = 2x − 1
Layer 2 (ReLU): h = max(0, z)
Layer 3 (output):  y = −h + 0.8
Safety propery:  y > 0   (for every x in [0,1])

The property claims the network's output is always positive. Is this true? Let’s see how the two verification methods handle this.

For constraint-based verification, its steps include:

Constraint-based methods exhaustively explore possibilities to find definitive answers but face scalability limitations with complex or large models.

For abstraction-based verification, its steps include:

The abstraction-based approach delivers speed and scalability by making safe estimates. While its initial findings can be imprecise, its ability to iteratively refine its focus makes it a powerful and practical tool for verifying large-scale systems.

The Elephant in the Room: Does it Scale?

The critical question always arises when discussing formal methods: Does it scale?

Historically, these techniques have carried a reputation for being powerful but difficult to scale, often seen as best suited for academic exercises or smaller models. This is a valid concern based on the field's origins.

But that is no longer the full story. This exact scaling challenge is what our team at Safe Intelligence has been dedicated to solving. By developing novel approaches that blend mathematical rigor with computational efficiency, we are making scalable verification a reality for the complex, production-grade AI systems being deployed today.

Putting Formal Verification to Work

Applying these advanced verification techniques enhances the reliability and safety of ML models in critical applications now. As you embark on your AI assurance journey, these are the core principles to guide you:

• Go Beyond Testing. Testing confirms behavior for inputs you've checked; formal verification proves correctness for entire classes of inputs you haven't, providing a far stronger guarantee of safety.

• Precisely Define Correctness. The foundation of meaningful verification is translating business needs into mathematical properties. Whether it's robustness, fairness, or consistency, a clear specification is essential.

• Verify Systems, Not Just Models. Compositional verification analyzes the complete system (the training data, the software environment, and its deployment dynamics) to build genuinely trustworthy AI.

In our upcoming articles, we’ll explore real-world case studies and practical applications that are helping make AI safer and more trustworthy across industries. But first, in the next article, we’ll show how we deploy these formally validated models in real systems. Stay tuned and stay safe.

Further Reading

• A Review of Formal Methods applied to Machine Learning

• Introduction to Neural Network Verification

• Algorithms for Verifying Deep Neural Networks

• Understanding Certified Training with Interval Bound Propagation

• Counter‑Example‑Guided Abstraction Refinement (CEGAR)

• Verification of Neural Networks Against Convolutional Perturbations via Parameterised Kernels

• Recomposition: A New Technique for Efficient Compositional Verification

Leave a comment

• TL;DR

• Why Benchmarking Matters

• What Counts as a "Benchmark"?

  • Infrastructure/System Benchmarks

  • Model Benchmarks

  • Data Benchmarks

• Building or Choosing Your Benchmarks

• Key Takeaways

• Further Reading

TL;DR

In the previous article of this series, we explored how to design effective test sets to get the best out of our validation exercise. But once you've nailed validation, the next big question is: How well does your model perform compared to other approaches or experiments? That's exactly where machine learning benchmarking comes into play. So, is this a feel-good exercise? Does it making you feel inadequate, or is it a useful tool? We will cover how to think of ML benchmarking as layers, cover choices between public and private benchmarking, and present a clear systematic workflow to connect technical model performance directly to measurable business impact. Without benchmarks, every experiment run thinks it’s the valedictorian.

On October 14, 1947, test pilot Chuck Yeager pushed his plane through the “sound barrier”, a limit many believed was fatal. His flight didn’t just produce a sonic boom; it shattered the assumptions of an entire field, proving that what seemed impossible was merely the next frontier. In computing, benchmarks are our sound barriers. They are the standard measures that motivate change, drive breakthroughs, and push researchers to exceed perceived limits. While your next developed model might not produce a sonic boom, it faces its own set of critical barriers. (Also, if you’re working on breaking the lightspeed barrier, let us know we’d like to invest!)

The previous article in the series gave us a foundation for model evaluation by designing test sets. Tests by themselves though are not anchored to anything. Benchmarking builds on tests by comparing multiple viable models. This comparison helps identify the best balance of performance, cost, latency, and risk for a specific business objective. Benchmarking can take time and be laborious but there are a lot of benefits. Let’s get started…

Why Benchmarking Matters

Benchmarking isn't about promoting one approach over another but objectively measuring and guiding requirements across any model deployment scenario. It fulfills four essential roles:

1. Objectively compare solutions: Benchmarks level the playing field. Rigorous benchmarks applied uniformly to models, infrastructure, and data methods cut through the marketing hype, internal biases, and subjective debates to deliver impartial rankings. However, as “Lies, Damned Lies, and Benchmarks” explains, public benchmarks can be gamed, as tweaking enough variables almost always yields a number that supports an argument, enabling deception and abuse.

2. Make Confident Trade-Off Decisions: Benchmarks provide clear success criteria, but rarely does one solution outperform others on every front. Instead, they reveal trade-offs. One model may offer higher accuracy but increase latency, while another cuts costs but struggles with out-of-distribution data. Teams can confidently choose the solution that best fits their operational needs using a multi-metric decision framework, such as a weighted scorecard.

3. Measure and Track Progress: Benchmarking isn't a single, isolated event. Regular, systematic benchmarking establishes historical performance records that help teams:

   1. Quantify incremental improvements.

   2. Quickly identify performance regressions.

   3. Demonstrate tangible ROI over time.

   Consistent benchmarking creates an evidence-backed narrative of continual improvement.

4. Raise the Bar and Advance the Field: Effective benchmarks set ambitious, motivating targets, pushing teams to innovate continuously. Much like Yeager's aviation milestone, benchmarking sets clear goals that inspire teams to explore beyond established limits, driving continuous improvement and field-wide advancement.

Organisations consistently apply benchmarks to ensure their machine learning solutions reliably deliver measurable value, regardless of the underlying technology trends.

What Counts as a "Benchmark"?

A benchmark is a controlled experiment applied to any layer of the ML stack. To isolate the impact of a change, you must hold other variables constant. Benchmarking in ML isn’t a monolithic activity but a practice that can be broken down into three primary layers: infrastructure/system benchmarks, model benchmarks and data benchmarks.

Infrastructure/System Benchmarks

• Goal: Identify the infrastructure (hardware and software) configuration that delivers reliable, scalable, and efficient model balancing performance, latency, throughput, cost, and energy use.

• Freeze: Model, dataset

• Vary: Hardware accelerators, precision modes (FP32 vs INT8), compiler optimization, deployment architectures, network topology for distributed training

• Metrics: Throughput (queries/sec), latency (p95/p99), cost per inference, energy use, CO₂ emissions

• Established Suites: MLPerf Training & Inference

• Real-world Example: Your baseline is a ResNet-50 model on NVIDIA V100 GPUs with 80 ms latency at $0.001 per inference. Benchmarking shows that migrating to H100 GPUs with INT8 precision reduces latency to 12 ms and cost to $0.0005 per inference, clearly justifying the upgrade.

Model Benchmarks

• Goal: Determine the model architecture and training strategy that maximises predictive power and aligns with business and ethical requirements. Model benchmarks are the traditional and most common type of benchmarks.

• Freeze: Dataset, system (infrastructure, including evaluation metrics)

• Vary: Model architectures, hyperparameters, feature engineering, and optimisation methods.

• Metrics: Accuracy, ROC-AUC, mAP, BLEU, risk, and operational readiness (covering robustness, explainability, fairness and bias)

• Public Yardsticks: ImageNet , CIFAR-10, GLUE, Kaggle/Zindi competitions

Real-world Example: A random forest baseline yields an AUC of 0.81 for churn prediction. A benchmark study reveals that a well-tuned LightGBM model lifts AUC to 0.87, surpassing both the baseline and the operational

Data Benchmarks

• Goal: Evaluate the ROI of data-centric strategies, quantifying how changes in data directly translate to model performance.

• Freeze: Model, system (infrastructure including evaluation metrics)

• Vary: Data volume, sampling strategies, annotation quality, synthetic data augmentation, data selection/valuation strategies.

• Metrics: Minority-class precision/recall, out-of-distribution robustness, performance uplift per dollar spent on data.

• Notable Suites: DataPerf

• Real-world Example: You have an image classification model with 75% accuracy. One benchmark evaluates the ROI of adding 50,000 synthetic images, which raises accuracy to 82%. Another evaluates the impact of spending the same budget on expert re-labelling of the 10,000 most-confused examples, which lifts accuracy to 85%. This data-centric benchmark provides a clear path for resource allocation.

Building or Choosing Your Benchmarks

A benchmark’s value is not in the numbers it produces but in the decisions it enables. Before any experiments are run, the first decision is whether to adopt a public standard or develop a private, internal benchmark. This choice dictates your point of comparison: the broader industry or your own unique business context. Most mature organisations use a hybrid approach, but understanding the trade-offs is key.

Once you’ve chosen your approach, define success with metrics that map directly to business impact. Use a portfolio of measures to capture trade-offs and ensure real-world relevance. With strategy and metrics set, focus on execution: rigorous benchmarking should be reproducible, objective, and decision-driven. Below is a practical workflow to guide each benchmarking run from hypothesis to actionable insight.

1. Formulate a Falsifiable Hypothesis: Start with a precise, testable question linked to business KPIs. Replacing our baseline logistic regression model with an XGBoost ensemble will lift ROC-AUC from 0.78 to 0.85, keep median inference latency below 10ms, and is projected to boost monthly revenue by £35k through a 12% reduction in false negatives.

2. Design the Reproducible Protocol: Meticulously document the "Freeze/Vary" framework, which is the blueprint for reproducibility and versioning.

3. Automate Execution and Logging. Integrate your benchmarks directly into the CI/CD pipeline and leverage experiment-tracking platforms like MLflow, Weights & Biases, or Vertex AI Experiments to automate execution. These tools capture every parameter, artefact, and result, ensuring a complete, auditable record for future analysis and reproducibility.

4. Analyse, Decide, and Communicate. Analyse results and tie your findings directly to the original hypothesis and business objectives. Summarise with a clear, evidence-based recommendation whether to ship, refine, or abandon, explicitly referencing both your starting hypothesis and the projected business impact.

Key Takeaways

Benchmarking can be time-consuming and sometimes a bit dispiriting if the numbers aren’t what you would like. However, it is worth the investment for production systems and will help drive incremental improvement.

• Treat benchmarking as a decision-making discipline to drive business impact, manage trade-offs, and de-risk deployments, not just as a metric-gathering task.

• Benchmarking isn’t limited to models. It applies across the entire ML stack. Evaluate infrastructure (hardware, deployment, efficiency), models (architecture, predictive power, robustness), and data (quality, quantity, coverage, ROI) to capture a complete, actionable view of system performance.

• Every benchmark should run through a disciplined loop, formulate a business-driven hypothesis, design a reproducible protocol, automate execution and logging, then analyse results to decide: ship, tweak, or kill.

• Build lasting trust by measuring performance, ensuring robustness, efficiency, fairness, and compliance, and guaranteeing full reproducibility through strict versioning and automated CI/CD checks.

For Large Language Models (LLMs), an excellent resource is TechTarget’s guide, “Benchmarking LLMs: A guide to AI model evaluation.” This can be complemented by exploring the Hugging Face Open LLM Leaderboard for regularly updated rankings of popular models.

While MLPerf, provides widely recognised standards for training and inference performance. For computer vision, benchmarks like ImageNet and COCO assess model capability, with newer benchmarks like Roboflow100-VL (robustness for vision-language models) is great for testing real-world robustness.

For tabular data, platforms like Kaggle and OpenML offer vast archives for establishing robust model baselines. Zindi complements these resources by providing fresh datasets that address specific business and societal challenges in Africa.

Further Reading

• Machine Learning System: Benchmarking AI

• Scientific machine learning benchmarks

• BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices

• 11 ML Performance Benchmarking Tools You Should Know

• Netflix: Designing the Netflix Prize

Leave a comment

• TL;DR

• Getting Test Set Design Right

  • Representativeness

  • Coverage

  • Probe for Model Failure

  • Maintain Strict Evaluation Boundaries

• Test Set Design Techniques

  • Simple Data Split

  • Cross-Validation Techniques

  • Bootstrapping

• What do evaluation metrics really tell us?

• Key Takeaways

• Further reading

TL;DR

Hey there, welcome back to the series. In the last post, we talked about why testing machine learning (ML) feels more like uncertain geology than precise geometry. This piece builds on that, arguing that real model validation—the process that makes a model truly ready for the real world—doesn't start with chasing a high performance score. Instead, it begins with deliberately designing your test sets. A test set is more than just the data left over after training. Real confidence comes from strategically engineering your test data to be a tough, real-world simulator that actively seeks out your model's breaking points.

No matter how advanced techniques become, every ML model remains inherently an approximation of reality. Recognising this essential limitation sets the context for why rigorous validation is not merely desirable but indispensable. This inherent imperfection raises the critical question: How good is good enough? The answer lies in building confidence, specifically, confidence that the model will reliably perform as intended in real-world operational conditions. This essential pre-deployment confidence is forged through validation.

Validation is a comprehensive assessment of all aspects of a model’s design, development, and final performance to ensure its readiness for deployment. Within this validation framework, two distinct technical processes are critical: testing and verification.

• Testing involves running inference on curated datasets to evaluate model performance against specific, predefined criteria in the scope of supervised learning.

• Verification, briefly introduced here but to be explored in detail in a later part of this series, uses formal mathematical methods to prove or establish high confidence in critical properties.

Before continuing, let's clarify our scope. In this series, we define validation as the technical, pre-deployment assurance process. While some include monitoring or governance under this umbrella, we treat them as separate concerns (post-deployment assurance). Validation focuses strictly on testing and verification that determine whether a model is ready for deployment.

Getting Test Set Design Right

A test set is not just a leftover slice of data, nor is its worth measured by size or a single performance score. Its fundamental purpose is to expose whether a model is truly deployment-ready by surfacing the necessary vulnerabilities. To achieve that, a well-designed test set rests on four pillars: representativeness of the production environment, exhaustive coverage of mission-critical use cases, targeted probes that hunt for hidden failure modes, and strict isolation from training data to preserve evaluation integrity.

Representativeness

A test set must resemble the world the model is about to enter so that its metrics give an unbiased, deployment-ready performance estimate. This demands disciplined sampling: include every region, population, or channel the model will encounter; balance rare but critical classes to avoid optimism that hides failure; and, when data are scarce, augment the training pool with synthetic examples so real-world instances can be reserved for evaluation. Furthermore, representativeness is temporal; a test set built on year-old data is a poor proxy for today's reality.

Coverage

Typical test sets evaluate model performance at discrete data points, leaving vast, unexplored regions around these points. Models may appear highly performant at isolated test points yet fail significantly when slight variations occur. Therefore, robust test design must encompass broader regions around key test points, ensuring performance stability across relevant data neighbourhoods.

A well-designed test set must go far beyond the “happy-path” cases where a model is already comfortable. It should deliberately sample the whole operational landscape, cutting across routine inputs and the rare, high-impact situations that can break a system. Work with domain experts to inject boundary conditions, edge cases, and infrequent but costly events (e.g., large fraud attempts). Slice scenarios by real-world stressors such as peak load and holidays—so you see how performance shifts when the stakes are highest. Only this breadth of coverage reveals whether the model is ready for production.

Probes for Model Failure

An effective test set must actively probe for model failures, not just confirm the model works. This requires designing specific inputs to uncover hidden weaknesses and logical inconsistencies. One approach is behavioural testing, where we verify the model's reasoning: does its output remain correctly unchanged when an irrelevant input is modified (invariance), and does it change predictably when a relevant one is adjusted (directional expectation)? The other approach is stress testing for stability, assessing performance on inputs with real-world imperfections like poor lighting, out-of-focus images, or noisy data. The most rigorous form of this is adversarial testing, which uses inputs carefully crafted to find a model's breaking points. If a test set never challenges the model, you’re not truly validating; it’s just performance theatre.

Maintain Strict Evaluation Boundaries

Isolation is non-negotiable; partition the test set at project inception, lock it away, and never let it contaminate training or model-selection steps. This absolute separation prevents all forms of data leakage, from direct sample overlap to subtle information bleed through proxy variables or shared preprocessing artefacts. Ultimately, this disciplined isolation is the only way to guarantee that the test set provides a true, unbiased measure of generalisation performance, ensuring you evaluate a model's ability to perform on unseen data, not merely its memorisation capacity.

Test Set Design Techniques

While the principle of isolation demands a strict separation of test data, the method of this separation is a critical design decision. A poorly chosen split can introduce high bias (underfitting), leading the model to oversimplify, missing important complexities in real-world data, or, conversely, high variance (overfitting), causing the model to memorise specifics rather than general patterns. Although it appears accurate during training, it performs poorly on new, unseen data.

Test set design is not an afterthought; when crafted efficiently, it drives the model toward genuine production readiness and gives stakeholders evidence they can trust. That rigour starts with how you carve raw data into a training slice for model building and an isolated slice for evaluation. Below are the principal splitting strategies:

Simple Data Split

A straightforward way to partition your data for training and evaluation is by using either a train–test or a train–validation–test split. In the latter, a validation set is specifically used for hyperparameter tuning, model selection, and iterative improvements, while the test set is held out until the very end for a final, unbiased performance check.

A common split ratio is 70% training, 15% validation, and 15% test. If you only need a high-level performance estimate, a 70–30 train–test split can suffice.

Cross Validation Techniques

While a simple data split is easy to implement, it can yield unstable or biased estimates, particularly if your dataset is small or not representative of the underlying distribution. Cross-validation overcomes these limitations by repeatedly training and evaluating the model on different subsets, offering a more robust view of how it will perform on unseen data. Crucially, every portion of the data is used for training and validation/testing at some stage, leading to more comprehensive performance metrics. Although cross-validation requires multiple training runs and can be computationally expensive—especially for large datasets—it often provides more reliable insights into real-world performance. It is especially valuable when you have limited data, allowing you to maximize both training and evaluation opportunities. Let’s look at some form of cross-validation:

k-Fold Cross-Validation

This cross-validation strategy splits the dataset into ‘k’ roughly equal ‘folds.’ In each of the k iterations, one fold is used as the validation/test set, while the remaining folds are used for training. Performance metrics are average over the k-folds for a more stable estimate

Repeated k-Fold Cross-Validation

This strategy involves performing k-fold cross-validation multiple times with different random splits. This approach further reduces variance in the performance estimate by averaging results across all runs. The ‘k‘ here doesn’t mean the number of folds but represents the number of times you repeat the entire training–validation process with different random splits.

Stratified k-Fold Cross-Validation

This strategy is a variant of k-fold cross-validation, where each fold is created in a way that preserves the overall class distribution. In simple terms, if you have an imbalanced dataset with the target of 60% negative, 30% positive, and 10% neutral, each fold will aim to have roughly 60% negative, 30% positive, and 10% neutral samples. ​​If you simply shuffle your data randomly, there is a chance that one fold may end up with more positive samples than another. Stratified k-fold tries to avoid that, as data imbalance can skew how you measure performance. Stratification ensures that every fold reflects the true proportions of each class (like a mini version of the full dataset).

Leave-One-Out Cross Validation (LOOCV)

Leave-one-out cross-validation (LOOCV) is a case of k-fold where k equals the number of data points (n). Each iteration uses one sample as the test set, while the remaining n-1 samples form the training set. A model is trained on these n − 1 samples and validated on the single held-out sample. This procedure is repeated n times—once for each data point as the test set—and the final performance metric is the average across all iterations. This maximizes your data for training, often leading to lower bias in the estimate. Training the model n times can be extremely computationally expensive, especially for large datasets. Leave-p-Out Cross-Validation (LpOC) generalizes LOOCV by leaving out p samples instead of one. This strategy requires training on all remaining data points for each possible selection of p samples, and it is still computationally intensive.

Monte Carlo (Repeated Random Sub-Sampling) Cross-Validation

Monte Carlo cross-validation creates multiple, independent train-test splits by repeatedly sampling the data at a fixed, predefined proportion (e.g., 70:20). For each split, a new model is trained from scratch and evaluated, with the final performance metric being the average across all iterations. While highly flexible, this method's primary trade-off is its lack of guaranteed coverage, as some data points may be tested multiple times while others are never selected for a test set at all.

Nested Cross-Validation

Using the same cross-validation procedure and dataset for both tuning and final evaluation can result in overly optimistic and biased estimates. Nested CV aims to prevent this by embedding one cross-validation loop within another to prevent information leakage between hyperparameter tuning and performance assessment. To avoid this, the outer loop splits the data into k folds, designating a fold(s) as the test fold and the remaining folds for training. Inside each outer training set, an inner loop performs its cross-validation purely for hyperparameter tuning, selecting the best model configuration. The tuned model is then evaluated on the outer test fold. This process repeats for each fold in the outer loop, and the results are averaged to provide an unbiased performance estimate. Although more computationally demanding, nested cross-validation is invaluable for reliably comparing models and hyperparameter settings.

Time series Cross-Validation

These cross-validation strategies are designed to respect the chronological order of data, ensuring that only past observations are used to predict future outcomes. Two main approaches are a sliding window and an expanding window. In the sliding window approach, a fixed-size window of the most recent observations is used for training, then the window shifts forward to discard older data and include new arrivals. This strategy keeps the model focused on recent trends but may lose potentially useful historical information. Conversely, the expanding window method starts with an initial set of observations and grows over time, retaining all prior data. While this strategy preserves the full historical context, older data might become less relevant, and the training set can grow unwieldy.

Time series cross-validation simulates real-world forecasting scenarios by providing multiple, chronologically consistent validation points. Additionally, the overlapping training sets can drive up computational costs, particularly for large datasets.

Bootstrapping

Bootstrapping is a resampling method for estimating model performance by repeatedly drawing new training sets with replacements from the original dataset. This strategy simply means each time you pick a data point for training, you place it “back” into the pool so you can choose the same point again in subsequent draws. Each “bootstrap” produces a dataset the same size as the original but with duplicated entries. The out-of-bag (OOB) samples—instances not chosen in that draw—serve as the test set. After multiple iterations, you average the OOB results to estimate performance. Bootstrapping, by contrast, may repeatedly sample certain points while omitting others, which can lead to a bias unless adjusted (e.g., via .632/.632+ methods).

What do evaluation metrics really tell us?

We often prioritise evaluation metrics like accuracy, precision, recall, F1 score, or RMSE when evaluating a model's health. See the Neptune AI guide on performance metrics for an excellent deep dive into how different metrics work and why they matter. However, it's crucial to remember these numbers are merely proxies—indirect measures of a model's true effectiveness in real-world scenarios. This is because direct measurement of real-world impact is often impractical during development; we optimise against these surrogate metrics. But there's an inherent risk: focusing too heavily on optimising these metrics can cause a model to "overfit" to metric specifics, exploiting quirks in training data rather than capturing meaningful generalisable patterns. This pitfall aligns with Goodhart's Law, which states:

Given these limitations, true confidence is forged not in a single score but in the rigour of the validation process itself. This robust approach begins with precise planning, which involves defining the scope of validation—from the data to the final inference pipeline—translating business objectives into explicit, measurable assessments and curating the immutable test sets needed for evaluation. This blueprint is then operationalized within a repeatable validation pipeline to ensure the same standards judge every model. For high-stakes applications, the process is fortified by independent review teams to eliminate confirmation bias, transforming validation from a one-time gate into an iterative cycle of continuously raising the bar for system safety and effectiveness.

Key Takeaways

• Validation is the entire pre-deployment assurance umbrella (testing and verification), while monitoring is strictly a post-deployment task.

• Design test data for representativeness, broad coverage (including edge cases), explicit failure probes, and airtight isolation.

• Evaluation metrics are proxies, not goals. Goodhart’s Law reminds us that over-optimising a single metric corrodes real-world usefulness; confidence comes from a well-governed, repeatable validation pipeline, not a solitary performance score.

• Choose the right split strategy to manage bias/variance trade-offs; pick deliberately rather than by habit.

Further reading

• Performance Metrics in Machine Learning [Complete Guide]

• Cross-Validation in Machine Learning: How to Do It Right

• Cross-validation: evaluating estimator performance

• Nested Cross-Validation for Machine Learning with Python

• Understanding Bootstrap Sampling: A Guide for Data Enthusiasts

Leave a comment

Executive Summary

During the first half of 2025, machine learning governance transitioned from principle to practice. Regulators translated guidance into enforceable rules, while organisations strengthened validation processes, formalised risk thresholds, and expanded transparency. In aviation and financial services, two of the most tightly regulated, safety-critical domains, ML systems have demonstrated measurable performance gains and reflect an industry that is striving for ever-greater reliability.

Operationalising ML Risk Management

Regulators moved decisively in the first half of 2025, turning guidelines into enforceable obligations and accelerating the maturity curve for responsible AI programmes. Some global regulators’ highlights include:

• EU AI Act (Europe) – First provisions in force (Feb 2) banning unacceptable‑risk AI and mandating AI literacy; full high‑risk obligations commence Aug 2 2025, covering data quality, documentation, risk & quality management, and EU database registration. Read analysis →

• Singapore Consensus on Global AI Safety (2025) – Introduces a defence‑in‑depth safety model spanning safe development, rigorous assessment (verification & validation), and ongoing control. Read analysis →

• United States – The NIST AI Risk Management Framework became the de facto national benchmark while states advanced their own laws. Read analysis →; Texas led with the Responsible AI Governance Act (TRAIGA), the first statute to ban certain high‑risk uses and launch an AI regulatory sandbox, as over a dozen other states draft similar bills. Read analysis →

• Japan – Parliament passed an innovation-first AI law establishing a cabinet-level AI Strategy HQ and voluntary guidelines to attract talent and investment while promoting responsible development. Read analysis →

• Kenya & Wider Africa – Kenya’s National AI Strategy 2025‑30 and similar initiatives across Africa combine ethical, inclusive, and innovation‑centric pillars to foster fintech‑driven growth. Read analysis →

• Gulf Cooperation Council (GCC) – Adopted “soft‑regulation” playbooks built on national AI visions and ethical charters, enabling fast innovation while binding enforcement remains light. Read analysis →

Sector Spotlight in High-Stakes Domains

Aviation: Ensuring Safety in the Skies

The aviation sector has effectively balanced innovation and safety by utilising advanced machine learning (ML) applications.

Finance: Risk‑Proofing Decisions

Financial institutions harnessed ML to enhance decision-making, strengthen compliance, and streamline operations.

Events and Conferences

The first half of 2025 featured numerous conferences and workshops focusing on trustworthy, safe and reliable ML. Here is a recap.

Looking Ahead (H2 2025 → 2026)

In the second half of 2025, we are excited to launch our new publication, Resilient by Design. This space is dedicated to exploring the art and science of building robust machine learning systems. You can expect technical insights, real-world use studies and community and collaboration. Subscribe for free, receive insights directly in your inbox, and become part of a growing community committed to making machine learning validated, reliable, repeatable, and robust.

Pull up a seat and join the conversation: Resilient by Design →

References

1. AeroTime: How aviation professionals can stay competitive in 2025

2. AirBus: Digital Twins: Accelerating aerospace innovation from design to operations

3. Appinventiv: How Digital Twin Technology is Transforming Airline Operations and Safety

4. AInvest: Joby Aviation's Dubai Milestone: A Catalyst for Urban Air Mobility's Mainstream Adoption

5. Machine Learning-Based Anomaly Detection in Commercial Aircraft

6. Applying artificial neural networks for multidimensional anomaly detection based on flight data monitoring during final approaches

7. The CAA’s Response to Emerging AI-Enabled Automation: Part A: Strategy for Regulating AI and Advanced Automation in Aerospace

8. Credolab: How Alternative and Traditional Data Work Better Together

9. Financial Conduct Authority: Credit where credit is due: how can AI's role in credit decisions be explained?

10. Silent Eight 360: 2025 Trends in AML and Financial Crime Compliance: A Data-Centric Perspective and Deep Dive into Transaction Monitoring

11. WJARR: Federated learning for privacy-preserving data analytics in mobile applications

12. Google Cloud: Google Cloud and Swift pioneer advanced AI and federated learning tech to help combat payments fraud

13. Bank of England: Financial Stability in Focus: Artificial intelligence in the financial system

14. LiquidityFinder: AI for Trading: The 2025 Complete Guide

Leave a comment

• TL;DR

• Why ML System Testing is Different: The Skyscraper vs. Rock Face Analogy

• A Deeper Look into ML Model Testing

• ML Model Testing Across ML Paradigms

  • Supervised Learning

  • Unsupervised Learning

  • Reinforcement Learning

• Key Takeaways

• Resources & Further Reading

TL;DR

Testing machine learning (ML) systems fundamentally differs from testing traditional code. It requires a multilayered approach that tests three artefacts: code, data, and models. Classic tests, such as unit and integration tests, are combined with specialised data, model, and end-to-end infrastructure tests. Performance metrics alone are insufficient; complementary tests are required. Each ML paradigm (supervised, unsupervised, and reinforcement learning) also needs distinct testing objectives and evaluation techniques.

Testing ML systems is hard due to their complexity across multiple dimensions—code, data, and models. Traditional testing methods, such as unit tests, integration tests, acceptance tests, system tests (functional and non-functional), regression tests, and practices like test-driven development (TDD), may feel tedious, but they provide a valuable, repeatable framework ensuring software correctness. In contrast, ML model testing primarily delivers performance statistics, which can sometimes feel ambiguous. What does 91% accuracy genuinely mean in practice?

Why ML System Testing is Different: The Skyscraper vs. Rock Face Analogy

Let's consider a visual analogy to put things into perspective: Imagine testing a glass skyscraper versus a fissured rock face. The skyscraper surface can be confidently verified with precise measurements and well-understood engineering principles. A handful of targeted tests are sufficient to justify your confidence. In contrast, a rock face is far more complex. Composed of possibly igneous, sedimentary, and metamorphic layers shaped by processes like sedimentation, pressure, erosion, heating, and cooling, its surface is inherently irregular and unpredictable. While you can take samples and apply statistical extrapolations, each measurement offers only an approximation, never absolute certainty. ML testing is much like the rock face.

Traditional software behaves predictably; the same input always yields the same output. ML systems, on the other hand, learn patterns from ever-changing data, creating opaque decision boundaries. A traditional software failure is typically straightforward: a button isn't clickable, or a form doesn't submit. Such deterministic logic makes debugging clear-cut. Conversely, an ML system's slight input variations (tiny perturbations) can trigger significantly different outputs or flip predictions. Let’s take a look at the comparison testing workflow for traditional software and ML systems below:

Here’s a comprehensive overview that clearly differentiates the types of testing:

The real cost of machine learning failures extends far beyond mere technical glitches, manifesting as significant financial losses, reputational damage, and even safety hazards. Consider the recent Zoox Robotaxi recalls, which highlighted the critical need for comprehensive testing; a crash in San Francisco revealed that insufficient pedestrian-detection tests forecasts forced Amazon's robotaxi unit to recall over 270 vehicles. Similarly, consider Zillow's iBuying collapse, where its ML models consistently overestimated home prices, resulting in a $420 million loss and the entire program being shut down. These examples underscore that rigorous, multifaceted testing is a good practice and an indispensable safeguard against catastrophic operational and financial repercussions.

A Deeper Look into ML Model Testing

Ensuring that ML models behave as intended—consistent and stable—is crucial, especially when they power critical real-world applications. A model's behaviour can vary significantly based on code, data quality, model architecture, training procedures, production environments, and even adversarial scenarios. The primary goal of ML system testing is to build confidence that the model will function correctly, robustly, and predictably under diverse, uncertain, and complex real-world conditions.

It is best practice to employ a multi-layered approach to thoroughly test our ML systems, which often includes:

• Data tests, which include, but are not limited to, schema validation, data quality assessments, statistical distribution checks, and outlier and drift detection.

• Model testing spans two critical dimensions: performance and behaviour. Performance testing evaluates how effectively a model meets its objectives through quantitative metrics and standard benchmarks, assessing its generalisation capabilities and detecting overfitting or underfitting. Behaviour testing shifts towards qualitative, scenario-driven assessments, examining how the model operates under challenging or specific conditions. This includes evaluating robustness to noise and adversarial attacks, interpretability (local and global explanations), compliance with safety constraints, and fairness measures like demographic parity and equal opportunity.

• ML infrastructure and end-to-end testing are crucial for operationalising ML systems. These tests span the entire MLOps lifecycle, including rigorous CI/CD pipeline checks, validating data integrity via feature store assertions, verifying the functionality of model registry and serving endpoints, establishing robust rollback mechanisms, configuring appropriate monitoring and alerting thresholds, and ensuring the correct execution of various deployment strategies.

ML Model Testing Across ML Paradigms

Just as different geological formations would require different assessment techniques, model testing's specifics for performance and behaviour must adapt to the model's learning approach, or its underlying paradigm. The three primary ML paradigms—supervised, unsupervised, and reinforcement learning—each have their difficulties and necessitate distinct testing approaches. Here's a table summarising the primary ML paradigms and their corresponding test objectives:

You can read more about these paradigms on the Wolfram blog, but here’s a quick overview is as follows:

Supervised Learning

Supervised learning is characterised by labelled input/output pairs; the main goal is to build a model that generalises well to unseen data. A good approach to testing is splitting the data into a training set, a validation set, and a test set. The training set is used to train the model, the validation set is used for hyperparameter tuning and model selection, and the test set is reserved for the final offline evaluation. Common splits include 70:15:15 (train:validation:test) or 70:30 (train:test). Cross-validation is a common technique that minimises the loss of training data for the validation set while still ensuring a rich test set by iteratively training and validating different subsets (folds).

Unsupervised Learning

Unsupervised learning aims to discover hidden patterns or groupings without labelled outputs, so there are no explicit splits of the dataset like we see in supervised learning. Instead, it makes use of intrinsic evaluation for clustering (like the silhouette coefficient, Davies-Bouldin index, inertia, the sum of squared distances, and the Calinski-Harabasz index) and dimensionality reduction (like reconstruction error and explained variance ratio). Additionally, it incorporates extrinsic evaluation methods, such as visualisations and domain expert evaluations, which are critical due to the lack of ground truth.

Reinforcement Learning

Lastly, in reinforcement learning, an agent interacts with an environment and learns a policy (a decision-making strategy) through repeated trials, aiming to maximise cumulative rewards and minimise penalties (negative rewards). Testing involves evaluating the learnt or trained policy in another controlled environment or set of episodes.

Key Takeaways

Without systematic tests, we have no insight into how a model behaves. Yet the tests we can run look different from classic software checks: they rely on statistical sampling and extrapolation rather than a tiny set of fully representative inputs. We're only just getting started with this series, but already, we're getting a hint that building ML systems and testing whether they will function well in the real world is a big challenge. This article offers a few key takeaways:

• Ultimately, effective testing supports ML systems' safe, effective, and responsible deployment. This is crucial for avoiding costly failures and building user trust.

• Different types of testing are essential, from data and model tests to infrastructure and end-to-end tests, ensuring comprehensive coverage of the ML system's components.

• ML model testing involves a blend of quantitative and qualitative model assessment. This means going beyond accuracy scores to understand model behaviour, such as robustness and explainability in real-world scenarios.

• We've seen how testing strategies vary for the respective ML paradigms (supervised, unsupervised, and reinforcement learning), each demanding specific objectives and evaluation techniques.

• The need for rigorous and continuous testing is paramount. Failure in ML is expensive; real-world incidents prove that weak tests translate directly into significant financial loss, reputational damage, and even safety hazards.

Resources & Further Reading

For further reading, see the links below, which are some of the best blog posts on testing for machine learning.

• Google: The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction

• How to Test Machine Learning Systems

• Testing Machine Learning Systems: Code, Data and Models

• Explaining and Harnessing Adversarial Examples

• Made With ML by Anyscale: Testing Machine Learning Systems: Code, Data and Models

• Neptune AI: Automated Testing in Machine Learning Projects [ABest Practices for MLOps]

• Train-Test Split for Evaluating Machine Learning Algorithms

• KDNuggets: Exploring Unsupervised Learning Metrics

• CS188 Artificial Intelligence, Fall 2013. Lecture 10: Reinforcement Learning

Leave a comment