How it works
Data poisoning is a way to compromise a machine learning system by altering or introducing data used in training, fine-tuning or evaluation. The goal can be to degrade overall performance or influence the model's behavior in specific situations. The problem can arise from a malicious action, a compromised source, or a data collection and labeling process without adequate controls.
The risk depends on where data enters the model lifecycle. A contaminated set during training can teach incorrect patterns. Fine-tuning data can change how the model responds to certain instructions. A compromised evaluation set can make a team believe the system works better than it really does.
In some targeted attacks, a small change is planned to affect a trigger or a class of inputs. However, not all bad data is an attack: duplicates, labeling errors, outdated data and out-of-scope samples also harm models. Analysis should separate malicious intent from quality failure, although both require mitigation.
Data poisoning is different from prompt injection. The first affects the data used to train or fine-tune a model, or an evaluation step. Prompt injection tries to influence a model during execution through the content it receives, for example in a page or document. The two risks can combine in systems that incorporate external data.
Practical example
Imagine a company that trains a classifier to separate legitimate messages from spam. If part of the messages is labeled incorrectly, the system can learn bad associations and send important messages to the wrong folder. If someone introduces deliberately manipulated examples, the impact can be more targeted.
A team can reduce risk by keeping a record of the origin and transformations of each dataset, reviewing suspicious samples and separating training data from evaluation data. Unusual changes in distribution, labels or performance deserve investigation. For critical data, use independent reviews and access control to the ingestion process.
After detecting contamination, it is not enough to delete a few lines without understanding the scope. Identify which versions of the set were used, which models were trained with them and whether re-evaluation or retraining is necessary. Record decisions and preserve evidence in a controlled way, without unnecessarily exposing personal data.
What it is for
The concept helps teams assess security and quality in data pipelines, internally trained models and solutions fine-tuned with customer data. It expands security review beyond code and infrastructure: provenance and integrity of data are also part of the risk surface.
Useful controls include trusted sources, schema validation, label review, versioning, audit trail, least privilege access and evaluation with data kept separate. No single control guarantees a dataset is free of manipulation, so combine technical checks with human review proportional to the application's impact.
When using a third-party model, ask how fine-tuning data is obtained and controlled. When building your own model, document who can change data and how incidents will be detected. The goal is to make suspicious changes identifiable before they are incorporated into system behavior.