Adversarial Machine Learning: Attacking and Defending AI/ML Systems

Attacking and Defending AI Systems

As organizations deploy machine learning models for security-critical decisions (fraud detection, malware classification, authentication), adversaries have developed techniques to manipulate these models.

Attack Categories

Evasion Attacks

Modify inputs to cause misclassification at inference time:

Image Classification: Adding imperceptible noise to an image causes misclassification:

# Adversarial perturbation
adversarial_image = original_image + epsilon * sign(gradient)
# Human sees: stop sign
# Model sees: speed limit sign

Malware Evasion: Modifying malware samples to evade ML-based detection:

• Appending benign content to change file features
• Modifying PE header fields
• Adding legitimate API calls to change behavior profile

Poisoning Attacks

Corrupt training data to influence model behavior:

• Label flipping: Change labels on training examples
• Backdoor insertion: Add a trigger pattern that activates specific behavior
• Data injection: Insert crafted examples that shift decision boundaries

Model Extraction

Steal model functionality through queries:

# Query the model API thousands of times
for sample in diverse_inputs:
    prediction = target_model.predict(sample)
    training_data.append((sample, prediction))

# Train a clone model
clone_model.fit(training_data)

Model Inversion

Recover training data from model outputs:

• Reconstruct faces from facial recognition confidence scores
• Extract text from language model probabilities
• Infer membership in training dataset

Defense Strategies

Adversarial Training

Include adversarial examples in training data:

for epoch in range(epochs):
    # Generate adversarial examples from current batch
    adv_examples = generate_adversarial(model, batch)
    # Train on both clean and adversarial data
    model.train(batch + adv_examples)

Input Validation

• Detect adversarial perturbations through statistical analysis
• Input preprocessing (smoothing, compression)
• Ensemble methods with diverse models

Model Hardening

• Differential privacy during training
• Model watermarking for theft detection
• Rate limiting and query monitoring
• Output perturbation to prevent extraction

Where SuperClue Fits

SuperClue does not test ML models for adversarial robustness. Its DAST scanner tests the web application and API that serve your model.