Your ML Training Pipeline Is Code. Treat It Like It.

The application code ships through a pipeline with peer review, static analysis, automated testing, and multiple approval gates before it reaches production. The machine learning training code that produces the models powering that application runs in a shared Jupyter notebook on a developer's laptop, sometimes copied into a cloud instance, checked into git occasionally when someone remembered, with credentials in the third cell.

Nobody has audited it. Nobody has treated it like the infrastructure it is.

This is the version of the supply chain problem that most security programs have not gotten to yet. The model is downstream of the training pipeline. Compromise the training pipeline and you compromise everything that runs on the model it produces, without necessarily leaving any trace in the model itself that standard evaluation will detect.

What the Training Pipeline Actually Is

A machine learning training pipeline is a sequence of code that pulls training data from somewhere, preprocesses and transforms it, feeds it through a model architecture during a training run, saves checkpoint files at intervals, evaluates the result against a held-out dataset, and packages the final model for deployment.

Each of those steps runs code. The code has dependencies. The dependencies come from somewhere. The training data comes from somewhere. The environment where all this runs has permissions - often substantial ones, because training requires access to production data, internal APIs, and cloud storage.

The security model most organizations apply to this is: nothing. The ML team is trusted. The notebooks are internal. The training cluster is behind the VPN.

Data Pipeline Compromise

The most accessible attack targets the data ingestion step. If the preprocessing stage pulls from a data source writable by more people than it should be, an attacker with access to that source can inject samples into the training data. The model learns from what it is fed. Poisoned training data can degrade model performance broadly, introduce backdoored behavior that activates on specific inputs, or shift the model's outputs in ways that serve the attacker without affecting benchmark scores.

This is harder to pull off against a well-managed data warehouse, and considerably easier against an S3 bucket with permissive IAM policies that the ML team reads from during nightly training runs. Most organizations have the former for their application databases and the latter for their data lake.

Training Script Injection

Script injection requires closer access than data poisoning, but the blast radius is larger and the results are more precise. A malicious pull request to the training codebase, a backdoored dependency in the requirements file, or a compromised account that pushes training scripts can embed arbitrary behavior directly in the training process. Unlike data poisoning, which requires the model to learn a behavior from examples, script injection can make the resulting model do exactly what the attacker wants.

The dependency angle is worth calling out specifically. Training pipelines tend to install from requirements files that nobody has reviewed with the same scrutiny applied to production application dependencies. A typosquatted package, a compromised package release, or a dependency pulled from a personal GitHub account instead of the official repository is enough to own the training run entirely.

Checkpoint Manipulation

Most long training runs save model state periodically as checkpoint files, because runs take hours or days and infrastructure is unreliable. A training process that resumes from a checkpoint that was modified between runs will continue training from a starting state the attacker controlled. The final model is downstream of that modified checkpoint, and the modification does not appear in the training logs that record what data was used.

Checkpoints are large binary files stored in object storage. They are rarely signed. Integrity is assumed, not verified. An attacker with write access to the checkpoint storage location can substitute a modified checkpoint that carries a backdoor, and the training run will resume from it without complaint.

Why Evaluation Does Not Catch This

Testing whether a model has been compromised at training time is harder than testing whether application software has been compromised. Standard model evaluation measures accuracy and performance against a held-out test set. A backdoored model performs normally on that test set. The backdoor activates on specific inputs the evaluation did not include, by design.

The practical consequence is that a compromised training pipeline may produce a model that passes every metric your team runs and still behaves incorrectly in production when an attacker provides the right input. You cannot evaluate your way to confidence in a training process you have not secured.

What Actually Helps

Version control with code review is the most basic control and the one most ML teams skip for code they consider "experimental." Training scripts, preprocessing code, and evaluation harnesses should be in source control, should require review before changes merge, and should have the same provenance tracking applied to application code. If it runs in your environment and produces something your application depends on, it is not experimental.

Training data provenance matters as much as code. The exact dataset that went into a training run should be recorded with a cryptographic hash. If you cannot reconstruct what data a model was trained on, you cannot investigate a suspected compromise after the fact, and you will find that you need to investigate exactly when it is most inconvenient.

Isolated training environments limit blast radius. An ML training cluster that cannot write back to the data lake it reads from, cannot reach production API endpoints, and cannot exfiltrate model weights to an external destination is substantially harder to use as a pivot point. The common pattern of training on infrastructure that also has production access is a configuration that makes lateral movement easy and auditing hard.

Checkpoint integrity verification is inexpensive and routinely skipped. Signing checkpoints at creation and verifying the signature before resuming from them costs almost nothing and eliminates the checkpoint manipulation attack entirely.

The Assumption Worth Updating

Most organizations with mature software security programs have not extended those programs to their ML infrastructure. The ML team developed separately, has different tooling, and has a different cultural relationship with the idea of security controls. The application security team reviews application code. Nobody reviewed the pipeline that produced the model the application depends on.

The model running in your production application went through a process. That process is code, infrastructure, and data. It has the same attack surface as any other code, infrastructure, and data. The fact that the end product is a neural network rather than a binary does not change what an attacker can do if they get access to the right step.

Your application code went through security review. Your model's origin story did not. That gap is worth closing before someone else finds it first.