Adversarial Machine Learning

What it is

Adversarial machine learning manipulates data or inputs to make a model behave incorrectly. Training-data poisoning influences learning, while evasion manipulates inputs at prediction time. These differ from ordinary accuracy errors and from conventional software exploits, although they can coexist. Practical impact depends on the model, access, task, and additional decision controls.

How it works

An attacker contributes misleading training examples or constructs inputs that exploit model sensitivities. Poisoning may target overall performance or a particular trigger; evasion aims to change a deployed prediction without changing model code. The attacker needs a relevant input or data path, not merely awareness that AI is used.

How people get tricked

Teams assume collected data is representative and trustworthy, or trust a model score as a final authority. Legitimate data drift can also cause errors, so suspicious output alone does not establish a deliberate attack.

Example scenario

A moderation classifier is deliberately shown altered content designed to be misclassified. A separate policy check and human review for consequential cases reduce reliance on that single prediction.

Warning signs

• Unexplained performance shifts tied to a data source or input pattern.
• Unexpected training-data changes or provenance gaps.
• Repeated targeted errors that differ from ordinary validation behavior.

If it happens to you

Preserve affected inputs and model versions, inspect data and pipeline changes, and separate malicious manipulation from drift. Retrain or roll back using trusted data where justified and reevaluate downstream decisions. Repair the data-access path that enabled the manipulation.