Domain 2 of 4 · Chapter 3 of 3

Adversarial AI Risk Mitigation

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing CompTIA SecAI+ premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Where the attack lands decides the defence
  • Training time: poisoning by objective
  • A model you did not train can arrive backdoored
  • Evasion: adversarial examples at inference
  • Direct prompting attacks and jailbreaks
  • Indirect prompt injection: no account required
  • Privacy attacks: what queries pull out of a model
  • Model extraction: rebuilding a model from answers
  • From answer to action: where an injection pays off
  • Testing the defence: red teaming and evaluation
  • Exam patterns: reading an adversarial-AI question

The four stages an attack can reach, and what changes at each

Question about the attackTraining timeInference inputInference outputDownstream action
What the attacker changesWhat the model learnsWhat the model readsNothing; the attacker readsWhat the answer sets off
Access neededTraining-data, model or source-code controlQuery access, or control of a resource the model ingestsQuery accessQuery access plus a wired-up tool or renderer
Named attacksAvailability, targeted, backdoor and model poisoning; supply-chain model poisoningEvasion and adversarial examples; direct and indirect prompt injection; jailbreakMembership inference, data reconstruction, property inference, training-data extraction, model extractionImproper output handling, excessive agency, exfiltration through a tool call
What it violatesAvailability or integrityIntegrity, or misuse for a jailbreakPrivacyIntegrity, and privacy when data leaves
Defence that blunts itData sanitization, robust training, provenance and hash verification, model inspectionAdversarial training and certified robustness for evasion; prompt formatting, instruction hierarchy and untrusted-data filtering for injectionDifferential privacy for training-data attacks; query limits and monitoring for extractionLeast-privilege tools, complete mediation downstream, human approval, output encoding
How you would noticeMetric collapse for availability poisoning; almost nothing for a backdoor until the trigger appearsAnomalous input distributions and repeated near-miss queriesSustained systematically varied queries from one identityUnexpected outbound calls and tool invocations the user never asked for

Decision tree

Did the attacker change the model before it was deployed? yes no Does the fault appear only with a trigger? Is the payoff a wrong answer, knowledge, or an action? yes no wrong answer knowledge an action Has accuracy fallen across all inputs? Perturbed sample, or written instructions? About the training data, or the model? Rendered or run, or did a tool fire? yes no sample instructions data model rendered tool fired Backdoor poisoning model inspection Availability poisoning metrics catch it Targeted poisoning sanitize the corpus Evasion adversarial training Prompt injection mark untrusted data Membership inference differential privacy Model extraction per-identity quotas Improper output handling: validate first Excessive agency least-privilege tools At every outcome: test the chosen defence with an adaptive attack
Cheat sheet

Cheat sheet

  • The stage an attack acts on decides which defences even exist
  • Name the violation: availability, integrity, privacy, misuse or supply chain
  • Black box is the realistic attacker knowledge, not the weak one
  • An open-weight model is a laboratory for attacking your closed API
  • Poisoning classes differ by how wide the damage is meant to be
  • A backdoor is the one attack needing training and test data control
  • Clean-label poisoning gives up label control and buys realism
  • A dataset shipped as a URL list can be bought out from under you
  • A pre-trained model's backdoor can survive your fine-tuning
  • An evasion attack changes the input, never the model
  • Three evasion defences have held up, and each charges for it
  • Detecting adversarial inputs is as hard as defending against them
  • An injection asks whose instructions win; a jailbreak asks whether refusal holds
  • Manual jailbreaks come in two families with different mechanics
  • A multi-turn escalation passes every per-message check
  • A single crafted document in a retrieval corpus can steer the answers
  • An injection can target availability and can spread itself
  • A rendered link can exfiltrate data with no tool call at all
  • Design the system as if the prompt injection will succeed
  • Membership inference leaks something even when nothing is reconstructed
  • Training-data extraction rewards the attacker who already knows something
  • Differential privacy protects the training data, not the model
  • Epsilon is a setting, so audit the privacy you actually got
  • Exact model extraction is impossible, and a functional equivalent is enough
  • Read a model-extraction attempt as reconnaissance, not just theft
  • Attack your own mitigation, and do it again later
  • Robustness trades against accuracy, and there is no best point
  • A model that is merely wrong is not an adversarial-AI problem

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2e2025) Whitepaper
  2. LLM04:2025 Data and Model Poisoning Whitepaper
  3. LLM03:2025 Supply Chain Whitepaper
  4. LLM01:2025 Prompt Injection Whitepaper
  5. LLM02:2025 Sensitive Information Disclosure Whitepaper
  6. LLM10:2025 Unbounded Consumption Whitepaper
  7. LLM05:2025 Improper Output Handling Whitepaper
  8. LLM06:2025 Excessive Agency Whitepaper