CY0-001 Cheat Sheet

The four terms are concentric sets, not synonyms. Artificial intelligence is the outer set, machine learning is the subset whose behavior is derived from data rather than written as rules, deep learning is the subset of machine learning built on neural networks with several layers, and generative AI is the family of deep-learning models that produce new text, code, images or audio instead of a label. Reading a claim about one of them as a claim about all of them is how vendor descriptions get over-generalised.

Automation is not a kind of AI, it sits beside the nested set. A playbook that blocks a source address after five failed logins executes a rule you can read, review and approve, so it behaves identically on every run; a model inferred its behavior from data, so it is probabilistic and shifts when the data shifts. The two compose freely, and most deployments mix them, which is why automated and learned are independent properties of a system rather than one label.

Trap Classifying a deterministic playbook as AI because the product name says so, then asking the vendor for model-evaluation evidence when the right evidence is the rule text and its change-control record.

NIST's AI Risk Management Framework defines an AI system as "an engineered or machine-based system that can, for a given set of objectives, generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments", designed to "operate with varying levels of autonomy". Two things follow for a practitioner. The output is a judgement that lands on a real or virtual environment rather than a stored answer, and autonomy is a setting on the deployment rather than a fixed property of the model, so the same model advising an analyst and the same model acting unattended need different controls.

Supervised learning trains a model to predict explicit, usually human-generated labels by comparing its predictions against known answers and adjusting its weights by how wrong they were. Malware, phishing and spam classifiers are the standard security examples. Because the labels define where the decision boundary falls, whoever influences the labels influences the model's verdicts, which makes where the labels came from and who could edit them a genuine assurance question rather than paperwork.

Trap Expecting a supervised detector to flag a category it holds no labelled examples of; it predicts the labels it was trained on, so an unrepresented class is outside what the model can express rather than something it merely misses.

Unsupervised learning trains on unlabelled data and finds structure in it, for example by clustering or grouping data points, which is what a learned traffic or behavior baseline is. The consequence is that the baseline is a description of the observation window and nothing more: activity that was present and unremarked while the model learned was taught to it as ordinary. Ask what window the baseline was learned from before trusting what it calls unusual.

Trap Assuming an unsupervised baseline encodes what good looks like; it encodes what the observation window contained, so anything routine during that window is now part of the norm.

Reinforcement learning trains a model to optimise its behavior according to a reward function by interacting with an environment and receiving feedback from it. Nothing in that loop reads your intent, so the reward you encoded is the goal the agent actually pursues. Whoever defines or influences the reward steers the resulting policy, which makes the reward definition the control point in exactly the way labels are the control point for supervised learning.

Trap Assuming an agent pursues the outcome described in the project brief; it maximises the number it was given, so a convenient proxy metric gets optimised instead of the goal.

A foundation model is trained on broad data using self-supervised learning, where the model creates its own objective from unlabelled data, and is then adapted to many downstream tasks, typically through fine-tuning or by being wrapped in an application. Because one such model is reused across many products, a flaw in it propagates to all of them at once. Treat the choice of base model as a supply-chain decision with a blast radius, not as an implementation detail of one feature.

Trap Assessing three assistants built on the same base model as three independent risks; they share whatever the base model gets wrong, so the shared component needs its own assessment.

Both are supervised tasks, and they differ in the shape of the answer. Classification predicts which of a set of discrete categories an input belongs to, such as a benign or malicious verdict on a file, while regression predicts a continuous numeric value, such as a numeric risk score. Naming the task type tells you what the output can and cannot support: a category needs a decision rule attached to it, and a number needs a threshold before it means anything operationally.

Most predictive models are discriminative, meaning they learn only a decision boundary between classes rather than a description of the data itself; logistic regression, support vector machines and convolutional neural networks are the usual examples. Generative models instead learn enough about the data distribution to emit new samples from it. NIST treats predictive AI and generative AI as separate families with separate attack taxonomies, so deciding which family a system belongs to is the step that selects the rest of the vocabulary.

Trap Assuming a generative model cannot perform a predictive task; NIST notes generative models are also used for predictive work such as sentiment analysis, so the output shape, not the model family, tells you what a given deployment is doing.

Deep learning is machine learning on neural networks with several layers, and a neural network is a stack of numeric transformations whose coefficients are the learned parameters. No individual layer carries a readable meaning, so there is no artifact to read the way you read a firewall policy. Assurance for a deep model therefore comes from testing its behavior, from the provenance of its training data, and from monitoring after deployment, rather than from inspection.

Trap Requesting a code review of the model as the assurance step for a deep-learning component; there is no readable logic in the parameters, so the evidence has to be evaluation results, data provenance and post-deployment monitoring.

NIST's adversarial machine learning taxonomy divides attacks by the stage they occur in, and placing an incident on that split names it faster than arguing over vocabulary. During the training stage an attacker who reaches the training data, its labels, the model parameters or the algorithm's code mounts a poisoning attack. During the deployment stage the model is already trained, so the attacker instead modifies inputs to change predictions, which is evasion, or probes the deployed model to infer information about its training data or its parameters, which is a privacy attack.

Trap Answering a poisoning report with input or prompt filtering; filters act on the request path and cannot repair a training set that is already corrupted, which needs data provenance, retraining and model rollback.

The model is the learned parameters, also called weights, plus the architecture that says how to apply them: training writes those parameters and inference only reads them. For generative systems the training stage usually consists of foundation-model pre-training followed by fine-tuning on task-specific data, so both halves sit on the training side of the split. A team that says it only fine-tuned has still produced a new artifact and owes you training-stage evidence: what data, what evaluation, what version.

Trap Filing a fine-tune as a configuration change because no model was trained from scratch; fine-tuning is further training, so the resulting artifact needs its own evaluation and provenance record before release.

NIST names attacker capabilities directly, and they map onto the pipeline: training-data control means inserting or modifying training samples, source-code control means controlling the learning algorithm's code, model control means modifying the model parameters, query access means sending inputs to a trained model and receiving its outputs, and resource control means modifying external documents or pages the model reads at inference time. Working out which of those the described attacker had resolves most terminology questions on its own, because each capability supports a different family of attacks.

Trap Assuming an attack that reached the model must have needed model control; query access alone is enough for evasion, prompt injection and extraction of information about the model.

A classifier emits a probability or a distance, and a person picks the number above which that value becomes an alert. Alert volume is therefore a configuration the security team owns, not a defect in the vendor's model, and the two error types move in opposite directions as the threshold moves: lower it and you catch more real problems while promoting more benign events into the queue, raise it and the queue shrinks along with coverage. The receiver operating characteristic (ROC) curve plots the true-positive rate against the false-positive rate across those thresholds, and the area under that curve (AUC) measures how well the model separates the two classes independently of any one threshold. Ask for the score distribution and the operating threshold, not just a headline accuracy figure.

Trap Suppressing or disabling a noisy detection to cut false positives; that converts a visible false-positive problem into an invisible false-negative one, where retuning the threshold and the surrounding logic keeps the coverage.

Comparing model output against ground truth gives four cells: a true positive is a real problem that was flagged, a false positive is an alert that incorrectly indicates a problem is present, a false negative is a real problem that was not flagged, and a true negative is a clean event correctly left alone. The false negative is the one to fear, because it leaves no artifact in the queue to tell you it happened, while a false positive at least announces itself and gets closed.

Trap Reading a high accuracy figure as good detection; benign events outnumber malicious ones by a wide margin in security data, so a model that calls everything benign still scores well on accuracy and detects nothing.

NIST's term for data collected at a different time and possibly under different conditions or in a different environment than the training data is out-of-distribution. When live traffic drifts that way the score distribution shifts even though the parameters are unchanged, so a threshold that was correct in one quarter is not automatically correct in the next. Drift monitoring is therefore part of operating a model rather than an optional extra, and it is the reason a model needs periodic re-evaluation where a rule does not.

Trap Treating a sudden change in alert volume as evidence someone tampered with the model; a shift in the input distribution moves scores on its own, so confirm the data before opening an integrity investigation.

Natural language processing (NLP) covers the tasks that operate on human-language text, from classifying a message to translating it to generating a reply. The technique underneath can be a keyword rule set, a classical supervised classifier, or a large language model, and that choice, not the label NLP, decides the risk profile. Ask which mechanism performs the task before deciding whether you are looking at a rules problem, a classifier problem, or a generative-model problem.

Trap Assuming natural language processing implies a large language model; a supervised classifier scoring email text is also NLP, and it fails and gets attacked in completely different ways.

A token is a piece of a sentence, usually a word, but often a subword for uncommon words, or a punctuation symbol, and text is converted into token identifiers before a model sees any of it. Model and context limits are expressed in tokens for that reason, and they do not convert cleanly into a character or word count, so capacity planning and truncation behavior have to be reasoned about in tokens.

An embedding is the condensed numerical representation a model produces from an input, and it is what similarity search actually runs on. Because it is derived from the source text, a collection of embeddings carries the sensitivity of the documents that produced it. Classify and protect a vector store by the material it was built from, the same way you would classify a derived index or an extract of a sensitive database.

Trap Treating a vector database as low-sensitivity because it stores numbers rather than documents; the vectors were computed from the documents, so the store carries their classification.

The system prompt, which is the application-specific instruction set the developer supplies in context, the user's message and any text the application fetched all arrive as one token sequence. NIST states the consequence directly: data and instructions are not provided in separate channels to the model, which lets an attacker use a data channel to inject instructions, a flaw it likens to SQL injection. Higher trust for the system prompt is an intention held by the developer, not a boundary the model enforces.

Trap Assuming the system prompt outranks whatever follows it because the developer wrote it; the model sees one undifferentiated sequence, so precedence has to be enforced outside the model if it is enforced at all.

A prompt injection exploits the concatenation of untrusted input with a prompt built by a higher-trust party such as the application designer, and the two variants differ only in who supplies the hostile text. A direct prompt injection is mounted by the primary user of the system through query access. An indirect prompt injection is mounted by a third party who can modify a document, page or record the system reads at inference time, and in that case the ordinary user is usually the victim rather than the attacker.

Trap Answering an indirect-injection scenario with stricter validation of user input; the hostile text arrived inside an ingested document, so nothing about the user's own message was ever malformed.

NIST uses confabulation for a generative system producing confidently stated but erroneous content, colloquially called a hallucination, and describes it as a natural result of the design: the model approximates the statistical distribution of its training data by predicting the next token rather than looking anything up. Generated text may also carry invented reasoning or citations that make a wrong answer more persuasive. The control is verification of the output against a deterministic source, not an integrity investigation.

Trap Opening a model-tampering investigation because a generated answer was confidently wrong with fabricated citations; with no adversary in the scenario that is confabulation, and only attacker-shaped content points at an injection or a poisoned model.

Retrieval-augmented generation pairs a model with a separate information retrieval system, finds material relevant to the user's query, and supplies it in context, so the usable knowledge changes with no retraining and access control stays in the retrieval layer where you can enforce it per user. Fine-tuning further trains the model on task-specific data and yields a new set of parameters, which suits a durable change in style, format or task behavior. Reach for retrieval when the content changes or has to be permissioned, and for fine-tuning when the behavior itself has to change.

Trap Choosing fine-tuning to give an assistant access to current, per-user-permissioned documents; the content ends up inside the parameters, where it cannot be refreshed daily or scoped to one reader.

An agent is a software program that interacts with its environment, receives information and takes self-directed actions in service of a larger, externally specified goal, and a language-model agent works by iteratively processing model output to act and feeding the results back as further context. That changes the failure mode: output is no longer text a person reads, it is an instruction something else executes, and every resource the agent reads on the way is another input channel into the same undivided context.

Trap Scoping an agent's risk to what it might say; the exposure is the set of external calls it is configured to make with values the model filled in.

Every control on an AI system attaches to one of four assets: the training data, the model artifact, the inference endpoint, or the prompt and response record. Reading a scenario for the asset first collapses four plausible options to one family, because data governance protects the corpus, integrity and provenance protect the weights file and its registry entry, authorization and guardrails protect the endpoint, and audit and retention protect the interaction record. Least privilege and key management are the two families that cut across all four rather than sitting under one.

Instructions such as "never reveal the connection string" sit in the same text channel an attacker can influence, so they steer a probabilistic model rather than enforce a policy. OWASP's LLM07:2025 System Prompt Leakage guidance says to avoid using system prompts to control behaviour where possible and to enforce security controls, especially privilege separation and authorization checks, in systems outside the model. The authorization decision belongs in application code the model cannot talk its way past.

Trap Hardening the system prompt in answer to a data-disclosure scenario; the prompt is recoverable text, and the control it was standing in for is an authorization check.

When a model can call functions, that tool credential is a separate identity and carries only the privilege the task needs. OWASP splits the failure, LLM06:2025 Excessive Agency, into three root causes with three different fixes: excessive functionality, where an extension that summarises mail can also send and delete it, fixed by a granular purpose-built extension; excessive permissions, where a database credential holds UPDATE and DELETE for a SELECT task, fixed by minimum permissions in the downstream system; and excessive autonomy, where a high-impact action runs unreviewed, fixed by human approval. Executing extensions in the individual user's context rather than under a generic privileged account keeps those fixes from being undone by a shared account.

Trap Reaching for an open-ended tool such as a shell wrapper because it covers future needs; it hands the model every capability the host has instead of the one the task requires.

A retrieval index assembled by a privileged crawler returns any document it holds unless the query itself is filtered by the asking user's permissions, so the check belongs in the retrieval layer at query time. OWASP frames the general rule as complete mediation, meaning the downstream system performs its own authorization check rather than trusting the layer that decided to ask. Skip it and one shared index makes every user a reader of every other user's documents.

Trap Relying on the application's front door to authorize the user and treating the index as internal; the front door authorized a session, not a document.

A value a model memorised is spread across weights rather than stored in a row you can delete, so the only remedies afterwards are retraining from a clean corpus or blocking the value on the way out. That makes minimisation, de-identification, and provenance pre-training controls, and everything applied later a compensating control. OWASP tracks the resulting exposure as LLM02:2025 Sensitive Information Disclosure, where a completion reproduces personal data the corpus contained.

Trap Answering a memorised-personal-data scenario with encryption of the model at rest; encryption stops someone who steals the file, not the model reproducing what it learned.

The dataset manifest names the source, the owner, the licence or contract permitting the use, the collection date, and the purpose the data was gathered for, one entry per source dataset. It is the artifact an auditor or an incident responder asks for, and it cannot be reconstructed after the fact, which is why an unrecorded dataset is rejected at intake rather than fixed later. OWASP's LLM03:2025 Supply Chain guidance puts vetting data sources alongside vetting models and packages, on the same reasoning: a corpus you cannot trace is a component you cannot vouch for.

Data minimisation, meaning the column never enters the corpus at all, is the strongest and cheapest de-identification approach and applies more often than teams assume, because a training task rarely needs the identifier itself. Pseudonymisation keeps per-subject grouping without the real identity, aggregation replaces rows with counts or statistics when the signal you need is a distribution, and differential privacy adds calibrated noise so an individual record cannot be reverse-engineered from the output, at a measured accuracy cost. Choose by what the task actually needs from the field, then apply the weakest technique that still satisfies it.

Masking systematically removes or obscures a field for display, such as showing only the last four digits of a card number, so it is a presentation-layer control and the stored record still holds the real value. Tokenisation replaces the value with a surrogate and keeps the original in a separate vault, recoverable by lookup under separate authorization, so it is the technique that actually takes the raw value out of the store. Both are legitimate controls; only one of them changes what a training corpus or a log file contains.

Trap Choosing masking to protect data at rest; masking is applied at presentation time and leaves the underlying record intact.

The default PyTorch checkpoint format is Python pickle, and Hugging Face's security documentation states that dangerous arbitrary code execution attacks can be perpetrated when you load a pickle file. The mechanism is the opcode stream replayed at load time: GLOBAL and STACK_GLOBAL import a Python object and REDUCE calls it with attacker-supplied arguments, so importing exec is enough to run anything. The safetensors format stores tensor data and a header with no mechanism to execute code on load, which makes the format choice a stronger control than any scan or loader default.

Trap Treating an antivirus scan as sufficient before loading third-party weights; signature scanning does not interpret pickle opcodes, and the pickle-import scanners that do are documented as best-effort.

OWASP's LLM03:2025 Supply Chain guidance is direct: use models only from verifiable sources, and use third-party model integrity checks with signing and file hashes, meaning the digest, to compensate for the lack of strong model provenance. Record the digest at intake so the serving process can re-check it when it loads the file, because a digest verified only at download proves nothing about the artifact sitting in the registry a month later. Do the verification in an isolated staging area holding no production credentials, so a load-time payload has nothing worth reaching.

OWASP asks for an up-to-date, accurate, signed component inventory in the form of a software bill of materials (SBOM), and points at OWASP CycloneDX as the emerging bill-of-materials format for AI components. The model card sits beside it and records the base model, a description of the training data, the intended use, and the known limitations. The pair turns the next vulnerability disclosure into a query instead of an investigation, and its scope includes fine-tuning adapters, quantised re-uploads, and the inference server's dependencies, not only the weights file.

SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models, is a community profile of the Secure Software Development Framework published in July 2024, and it names three audiences: producers of AI models, producers of the systems that use those models, and acquirers of those systems. Downloading or purchasing a model therefore brings an organisation into scope for the profile's practices rather than exempting it.

Trap Assuming the generative-AI secure-development profile applies only to organisations that train their own models; the acquirer is named as an audience explicitly.

Managed guardrail services evaluate both the input prompt and the model completion, because the two directions catch different failures. What arrives is attacker-influenced text needing content filtering, topic denial, and size limits; what leaves is shaped by that same text, so it needs its own pass before anything downstream sees it. Decide the failure mode explicitly as well: a guardrail that fails open leaves the endpoint unprotected exactly when it is under load.

The same families recur across implementations: content filters that score harm categories such as hate, insults, sexual, violence, misconduct, and prompt attack with a configurable strength per category; denied topics defined for the application's own context; word filters holding exact-match blocklists; sensitive information filters that block or mask personally identifiable information (PII), with regular-expression patterns as an extension; and contextual grounding checks that flag a response not grounded in the retrieved source. Amazon Bedrock Guardrails and Azure AI Content Safety are two implementations among others, and the grounding check exists only in retrieval-augmented designs, because it needs a retrieved source to compare the answer against.

OWASP's LLM05:2025 Improper Output Handling covers insufficient validation of model output before it reaches another component, and its named consequences are ordinary application vulnerabilities: cross-site scripting when output is rendered in a browser, SQL injection when a generated query runs unparameterised, path traversal from a constructed file path, and remote code execution when output reaches a shell. Its first prevention strategy is the line to keep: treat the model as any other user, adopt a zero-trust approach, and validate its responses. Concretely that means context-aware encoding chosen for the sink, parameterised statements for anything touching a database, a Content Security Policy on pages that render completions, and schema validation where structured output was requested.

Trap Fixing a rendered-script incident by strengthening the system prompt; the model produced exactly what the attacker asked for, and the missing control sits downstream of it.

A content filter returns a judgement at a configurable strength and will occasionally be wrong in both directions, so it belongs where only a model can judge: tone, topic, and harm. A schema check, an allowlist, and a parameterised query either hold or fail loudly, so they belong wherever the correct shape can be defined. Use both, and remember that neither is an authorization control: a filter decides whether text is acceptable, never whether this caller may have it.

OWASP's LLM10:2025 Unbounded Consumption gathers three harms behind one control surface: denial of wallet, where ordinary-looking volume drains a pay-per-use budget; model extraction, where enough query and response pairs train a functional equivalent of your model; and resource depletion from deliberately expensive inputs. Extraction needs many queries, so a per-identity quota raises the attacker's cost far more than a global endpoint limit they can hide inside, and a per-IP limit alone spreads across sources while penalising shared corporate egress addresses. Pair the quota with input size limits, request timeouts, throttling, and anomaly detection over query patterns.

Trap Answering a sudden inference cost spike by adding capacity or enabling autoscaling; that removes the symptom and enlarges the bill the attacker is running up.

One platform-default key covering the training corpus, the model registry, and the prompt and response record cannot be revoked for any one of them without breaking the other two, which is the whole argument for a key boundary per asset. Customer-managed keys, sometimes offered as bring-your-own-key, supply the properties a question is usually testing: create, rotate, disable, and revoke, plus an audit trail of which key protected what. Cover the storage locations teams forget as well as the obvious ones, particularly training checkpoints written to scratch storage and the vector store built from the corpus.

OWASP's LLM07:2025 System Prompt Leakage guidance asks you to avoid embedding sensitive information such as API keys, authentication keys, database names, user roles, and permission structures in prompts, and to externalise that data to systems the model cannot access. In practice the application fetches credentials from a secret store at call time under its own workload identity, tool credentials are issued to the tool layer rather than described to the model, and connection strings never appear in a prompt template. The same rule reaches backwards into training data, where a credential committed to a code corpus becomes a memorised string a completion can reproduce.

Trap Storing an API key in the system prompt on the reasoning that users never see it; the system prompt is recoverable text, which is exactly what LLM07 names.

An inference log carrying only a timestamp and a token count cannot answer what the system said, to whom, and under which policy. Record the request identifier, the per-user caller identity rather than the shared application account, the model identifier and version, the guardrail policy version and its verdict, the tool calls attempted, approved, and refused, the unit counts, and the prompt and completion content. Write it to an append-only store the application identity cannot rewrite, because if the identity that produces the evidence can also edit it, compromising the application compromises the record.

The deletion path has to reach every derived copy: the training corpus, the embeddings built from it, and the inference logs that captured whatever a user typed into the prompt. OWASP's LLM02 guidance pairs clear data retention and deletion policies with a route for users to opt out of having their data included in training, and the retention period is set per source in the same manifest that governs the corpus. Keep the shortest period the obligation allows, because a log store holding customer conversations is a more attractive target than the endpoint that produced them.

A guardrail policy edited live is a security control with no change history, so bind a specific policy version to the endpoint and ship a policy change the way you ship code. Managed guardrail services are invoked by identifier and version for exactly this reason, which also lets a bad policy be rolled back without touching the model. The version then belongs in every log entry, so a later review can tell which policy was in force when a given completion was allowed.

Four things produce output in an AI-assisted detection stack, and each fails in its own way: a signature or rule match fires on a cataloged known-bad pattern and misses variants; an anomaly baseline fires on activity that is rare for that entity and produces benign-but-unusual false positives; a supervised classifier fires on resemblance to labelled past cases and decays as those labels age; and a large language model produces no verdict at all, only summaries, queries and drafts. Knowing which one fired tells you in advance how the alert is most likely to be wrong and what evidence will settle it.

Trap Treating an assistant's written conclusion as a fourth kind of detection; it compares nothing against anything and holds no detection state, so it can only restate what you handed it.

Signature-based detection compares observed events against cataloged patterns of known bad activity. Anomaly-based detection compares them against learned profiles of normal behaviour for users, hosts, network connections or applications. Stateful protocol analysis compares them against vendor-supplied universal profiles of how a protocol should and should not be used, tracking session state rather than single packets. Machine learning operates inside the second one; a real product usually runs more than one methodology at once.

Trap Filing stateful protocol analysis under anomaly detection because both look for deviations; its profiles are vendor-supplied and universal to the protocol, not learned from your own hosts and network.

A baseline deviation says the activity is unusual for that entity, and the detector has no access to whether it was harmful. NIST SP 800-94 notes that anomaly-based products often produce many false positives because benign activity deviates significantly from profiles, especially in diverse or dynamic environments. The corroboration that turns rarity into a case has to come from elsewhere: a second signal, an enriched context, or a rule that encodes known-bad behaviour.

An anomaly profile is generated over an initial observation window, typically days and sometimes weeks, called the training period. Legitimate activity that did not happen inside that window is later read as a significant deviation, which is why quarterly closes, monthly maintenance jobs and annual export runs alert the first time they occur. The fix is to document and tune the specific exception once it is confirmed benign, and to make sure the rebuild window covers a full business cycle.

Trap Widening thresholds globally to stop one recurring maintenance job alerting; that lowers sensitivity for every entity instead of recording a single known exception.

SP 800-94 calls inadvertently including malicious activity in a profile a common problem with anomaly-based products, because the sensor may observe an attacker while it is building its initial profiles. The result is a detector that is quietly blind to exactly the behaviour you most need to see, and it looks healthy from the outside because it is not alerting. Where administrators can edit a profile, excluding activity known to be malicious is the remedy; otherwise the profile has to be rebuilt from a window you can vouch for.

Trap Assuming a quiet training window was a clean one; low alert volume during profiling is equally consistent with an attacker operating below the thresholds being learned.

A static profile is unchanged once generated unless the product is told to rebuild it, so SP 800-94 warns it will eventually become inaccurate as systems and networks change and needs periodic regeneration. A dynamic profile adjusts constantly and avoids staleness, but is susceptible to evasion: an attacker who performs a small amount of malicious activity and then increases the frequency slowly enough can have that activity absorbed into the profile as normal. Because the two failures pull opposite ways, mature deployments run a learning baseline alongside rules that do not learn, so slow escalation still hits something fixed.

Trap Choosing a dynamic profile to solve stale baselines and considering the problem closed; it trades staleness for the slow-and-low evasion, which is the harder failure to notice.

The joint CISA and NCSC guidelines for secure AI system development make measuring the outputs and performance of the model and system a standing duty of the operate-and-maintain stage, precisely so that both compromise and natural data drift show up as observable changes in behaviour. SP 800-94 gives the operator-side version: baselines collected for anomaly-based detection should be rebuilt periodically to support accurate detection. In practice a rebuild is triggered by a measured drop in precision or recall, a deliberate environment change that invalidates the old normal, evidence that the training window overlapped an incident, or a scheduled interval that exists so nobody has to notice.

Trap Reading a drop in alert volume as good news; a detector that has drifted into silence is indistinguishable from a quiet week until you measure it against known outcomes.

A supervised classifier scores new events against labelled examples of past benign and malicious ones, so it cannot start until you have a curated set of closed cases with correct verdicts on both sides. Where that history does not exist, the working answer is rules plus baselining while analysts build the label set as they close cases. The labels are also the maintenance burden: as the attack landscape and the environment move, a classifier trained on an older label set decays silently, still scoring confidently against a world that has changed.

Trap Training on alerts analysts bulk-closed without recording why; those labels encode the queue's triage habits rather than ground truth, and the model learns to reproduce them.

Precision is the ability of a model to avoid labelling negative samples as positive, computed as true positives over every alert raised, so it is the fraction of the queue that was worth opening. Recall is the ability of a model to detect all positive samples, computed as true positives over every real attack, so it is the fraction of intrusions actually caught. They trade against each other through one knob, the decision threshold: lower it and recall rises while precision falls and the queue grows; raise it and the reverse happens. Publish both, at the threshold you actually run.

Trap Reporting accuracy as the headline metric; when genuine attacks are rare, a detector that alerts on nothing scores almost perfect accuracy while catching nothing at all.

When real incidents are vanishingly rare relative to event volume, precision is governed far more by the false-positive rate than by recall. Take an illustrative day of 2,000,000 authentication events holding 20 real malicious ones: a detector with 90 percent recall and a 0.1 percent false-positive rate catches 18 of them and raises about 2,000 false alerts, so roughly one alert in 112 is real. Cutting the false-positive rate tenfold, to 0.01 percent, still leaves about 200 false alerts for those 18. The numbers are arithmetic on assumed inputs rather than any product's benchmark, but the shape holds for every rare-event detector.

Trap Dismissing a 0.1 percent false-positive rate as negligible; against millions of benign events per day it is thousands of alerts, which is the whole shift.

The answer to alert fatigue is alert tuning: refining correlation rules, suppressing patterns confirmed benign, and adjusting thresholds so genuine events stand out. Disabling the detection removes its false positives and its true positives together and leaves a permanent blind spot that no other control is covering. Tuning is also not free in the other direction, because pushed hard enough to eliminate false positives it starts producing false negatives, so it is a standing balance rather than a one-time threshold drop.

Trap Raising the threshold until the backlog clears without measuring what stopped firing; the alerts you no longer see become false negatives that nothing is counting.

Scoping a detector to privileged identities, service principals, or hosts holding regulated data raises the proportion of genuine incidents in the events it sees, which lifts precision without touching the model at all. Fewer events with a higher genuine-incident rate is a strictly better queue than more events at the same false-positive rate. Reach for broad coverage when the question is what you can reconstruct after the fact, and for narrow scoping when the question is what an analyst opens tonight.

Trap Onboarding more log sources to fix a noisy detector; extra benign volume at an unchanged false-positive rate makes the queue worse, not better.

Calibration is a specific and measurable property: a well-calibrated model correctly classifies 100 percent of the predictions it assigns 100 percent confidence, 50 percent of those it assigns 50 percent confidence, and so on. Most detection scores are not calibrated, so the number is a ranking device that says one event is stranger than another and nothing more. That is entirely adequate for ordering a queue and useless as a probability you can act on.

Trap Reading a risk score of 87 as an 87 percent chance of compromise, then setting an automation gate at that number as though it were a confidence level.

NIST SP 800-61r3 names the factors to weigh when prioritising: asset criticality, functional impact of the incident, data impact of the incident, stage of observed activity, threat actor characterisation, and recoverability. Anomaly magnitude is not among them, because strangeness and consequence are different quantities. A useful ranking model consumes the anomaly score as one feature and weights it against those factors; if the tooling can only sort by score, the ranking is still the analyst's job and the model has moved work rather than removed it.

Trap Sorting the queue by model score alone; a genuinely weird but harmless event then outranks a routine-looking privileged action on a critical asset.

Correlation groups the alerts that belong to a single attack so six signals about one intrusion become one case rather than six. It is also the cheapest precision available, because a pattern spanning several stages of an intrusion is a far stronger claim than any one anomaly inside it. Products that do this automatically attach the timeline of alerts and underlying raw events, the tactics observed, the users and devices involved, and the supporting evidence, which is exactly the context a triage decision needs.

Trap Counting correlated alerts as independent confirmation; several alerts derived from the same underlying event add volume to the case without adding evidence.

SP 800-94 observes that analysts often cannot determine why a particular anomaly alert was generated or validate that it is not a false positive, because of the number and complexity of the events behind it. Enrichment answers that by attaching the facts the analyst would otherwise fetch by hand: asset owner and criticality, the identity's normal peer behaviour, reputation or threat intelligence on the observed indicator, and what changed recently. The detector still will not say why it fired, but the analyst can now see the answer in one place instead of five consoles.

Trap Expecting a model explainability feature to replace enrichment; feature attributions say which inputs moved the score, not what the entity was actually doing at the time.

What decides whether a detection verdict may fire an action by itself is not the model's confidence but what the action costs when the model is wrong, which comes down to reversibility and blast radius. Revoking a session token, forcing re-authentication, quarantining one endpoint or blocking a single indicator on one host all fail cheaply. Isolating a production segment, disabling a privileged service account, re-imaging a host or deleting suspected artefacts turn a false positive into a self-inflicted outage, so those need a named human decision on the record, plus an owner and an expiry for any containment that gets applied.

Trap Using a higher confidence threshold as the gate for an irreversible action; raising the bar changes how often you are wrong, not what being wrong costs.

Rebooting, re-imaging or wiping a host removes volatile evidence, and cutting an attacker's channel can end an observation that was about to establish scope. SP 800-61r3 treats the mirror-image choice the same way, warning that deliberately redirecting an attacker to a sandbox to gather more evidence delays containment and eradication and should be discussed with the legal department first. The resolution is sequencing rather than picking a side: an automated action that captures memory, snapshots disk or exports the relevant logs before it isolates the host keeps both options open.

Trap Re-imaging the affected host as the first containment step; it is fast and clean and it destroys the volatile evidence that would have proved how far the intrusion reached.

NIST defines confabulation as the production of confidently stated but erroneous or false content, known colloquially as hallucination, and describes it as a natural result of how generative models work rather than a defect to be patched out. The part that catches experienced analysts is that outputs may include confabulated logic or citations that appear to justify the answer, and models sometimes lay out plausible steps even when the conclusion is wrong. Check the conclusion against the underlying log record, never against the explanation attached to it.

Trap Accepting an assistant's finding because it showed its working; the reasoning chain is generated by the same process as the answer and can be fabricated alongside it.

NIST describes automation bias as excessive deference to automated systems, arising as people come to over-rely on generative output or perceive it as higher quality than other sources, and notes that it makes confabulation risk worse rather than sitting beside it. It does not announce itself, because a shift where nobody questioned the tool looks like a fast, tidy shift. The countermeasures are procedural: keep high-impact actions behind a named approver, record who approved each one, and review a sample of auto-closed cases on a schedule so the claim that the model handled it is periodically tested.

Trap Relying on analyst experience as the safeguard; deference grows with the system's apparent reliability, so the more accurate the tool gets the weaker that control becomes.

SP 800-61r3 defines evidence as grounds for belief or disbelief, the data on which proof is based, and treats collected incident data as evidence even when formal chain-of-custody handling is not used, to be retained under the organisation's evidence preservation and data retention policies. A model's summary is a derived artefact, so it belongs in the case notes and never in the evidence position of a report, a regulatory notification or a legal process. Keep the prompts and outputs too, since the CISA and NCSC secure operation guidance calls for logging system inputs such as queries and prompts to support audit, investigation and remediation.

Trap Letting the underlying records age out because the assistant's summary was archived; the derived note is not the artefact the retention and preservation policy covers.

Filenames, command lines, user-agent headers, e-mail subject lines, commit messages and HTTP paths all carry strings an adversary chose, so feeding raw telemetry to an assistant hands that adversary a writing channel into the prompt. OWASP catalogues this as LLM01:2025 Prompt Injection in the 2025 Top 10 for LLM Applications. The operational consequence for detection and response is narrow and firm: output derived from attacker-controlled text must never drive an action until a human has read the underlying record.

Trap Treating log content as trusted because it came from your own systems; the system recorded the event faithfully, but the field contents were written by whoever caused it.

Four classes of instrument sit behind AI obligations, and the class predicts the consequence better than the document's title does. A binding regulation such as the EU AI Act or the General Data Protection Regulation (GDPR) is enforced by public authorities with administrative fines and, in the AI Act's case, removal of the product from the market. A certifiable standard such as ISO/IEC 42001 is enforced by an accredited certification body, and failure costs you the certificate. A voluntary framework such as the NIST AI Risk Management Framework carries no penalty at all, and government guidance carries less still. Only the first class makes "we decided not to" indefensible; the other three are choices you can justify on cost, timing or scope.

Article 2(1) applies the Act to providers placing an AI system on the Union market or putting it into service in the Union irrespective of whether they are established in the Union or in a third country, to deployers established in the Union, and to providers and deployers in a third country where the output produced by the system is used in the Union. Where the model runs and where the company is incorporated are not part of the test. The GDPR uses a different but similarly extraterritorial hook, following the personal data of people in the EU wherever the processing happens, so a US-hosted model serving EU customers is inside both regimes at once.

Trap Concluding that a US-hosted deployment is out of scope because no servers or corporate entity sit in the EU; the output-used-in-the-Union limb of Article 2(1) catches it anyway.

The tiers are unacceptable risk, high risk, transparency risk and minimal risk, and the classification is a property of the system and its intended purpose rather than of the company that bought it or the size of the model. Unacceptable-risk practices are banned outright, high risk carries the heavy conformity and documentation regime, transparency risk carries disclosure duties, and minimal risk carries no tier-specific obligation. Fix the tier first in any scenario, because it decides whether you are reading a prohibition, a heavy compliance programme, a disclosure requirement, or nothing at all.

Article 5(1) of the Act as adopted lists eight prohibited practices, points (a) to (h). They include social scoring that leads to unjustified detrimental treatment, untargeted scraping of facial images to build facial-recognition databases, inferring emotions in the workplace and in education institutions, biometric categorisation to deduce traits such as race or religious belief, and predicting criminal offending based solely on profiling. The Digital Omnibus on AI adds a further prohibition, on generating non-consensual intimate or sexual content and child sexual abuse material, once that amending regulation applies. What separates this tier from every other is that no control set makes the practice acceptable: documentation, human oversight and a risk assessment do not open a path. The only compliant answer in the Union is not to build or use it.

Trap Offering an impact assessment, human oversight or thorough documentation as the way to deploy a prohibited practice; those satisfy high-risk duties, not a prohibition.

Under Article 6 a system is high risk either because it is a safety component of, or is itself, a product covered by the Annex I Union harmonisation legislation that requires third-party conformity assessment, or because it falls in one of the eight Annex III areas: biometrics, critical infrastructure, education and vocational training, employment and workers management, access to essential private and public services, law enforcement, migration and border control, and administration of justice and democratic processes. Credit scoring sits in the essential-services area and resume screening in the employment area, which is why ordinary business systems land here so often.

Trap Reading Annex III as the whole definition of high risk; an AI safety component inside a regulated product reaches the same tier through Annex I without appearing on the Annex III list.

Article 6(3) lets an Annex III system escape the high-risk classification when it does not pose a significant risk of harm and it meets one of four conditions: it performs a narrow procedural task, it improves the result of a previously completed human activity, it detects decision-making patterns or deviations without replacing or influencing a completed human assessment, or it performs a preparatory task to an assessment. The same provision then closes the door on systems that perform profiling of natural persons, which stay high risk regardless of the four conditions. Check the profiling question first, because it makes the rest of the analysis moot.

Trap Claiming the narrow-procedural-task condition for a candidate-ranking or scoring model; evaluating personal aspects of people is profiling, which keeps the system high risk.

Article 50 requires providers to tell people they are interacting with an AI system unless that is obvious to a reasonably well-informed observer, and to mark synthetic audio, image, video and text in a machine-readable format detectable as artificially generated; deployers must inform people exposed to emotion-recognition or biometric-categorisation systems and disclose deep fakes. These duties trigger on what the system does, so a high-risk system that chats with customers owes them on top of its high-risk obligations. The tiers partition the heavy obligations, not every obligation.

Trap Reading transparency risk as a tier that excludes the others, so a high-risk system with a chat interface is treated as exempt from the Article 50 AI-interaction disclosure.

A deployer uses an AI system under its own authority outside a purely personal, non-professional activity, and Article 26 sets out concrete duties for a high-risk system: use it in line with the provider's instructions, assign human oversight to people with the necessary competence, training and authority, keep controlled input data relevant and sufficiently representative, monitor operation and suspend and notify when use presents a risk, retain the automatically generated logs for at least six months, inform workers' representatives and affected workers before workplace deployment, inform people subject to the system, and cooperate with competent authorities. Article 26(9) also obliges the deployer to feed the provider's information into its own GDPR impact assessment, which is the seam where the two regimes meet.

Trap Answering conformity assessment, technical documentation or EU-database registration for an organisation that simply uses a bought system as supplied; those are provider duties under Article 16.

Article 25(1) names three circumstances that convert a distributor, importer, deployer or other third party into the provider of a high-risk system: putting their name or trademark on a system already placed on the market, making a substantial modification that leaves it high risk, or modifying the intended purpose of a system, including a general-purpose AI system not previously classified as high risk, so that it becomes high risk under Article 6. The consequence is not incremental: a short Article 26 checklist is replaced by conformity assessment, technical documentation, a quality management system and registration.

Trap Assuming only engineering changes can flip the role; a name or trademark alone triggers Article 25(1)(a) with no change to the system at all.

Article 99 sets three ceilings, each expressed as an amount or a share of total worldwide annual turnover for the preceding financial year, whichever is higher for an undertaking. Breaching the Article 5 prohibitions draws up to EUR 35 000 000 or 7 %. Breaching most other operator obligations, including provider and deployer duties, draws up to EUR 15 000 000 or 3 %. Supplying incorrect, incomplete or misleading information to notified bodies or national competent authorities draws up to EUR 7 500 000 or 1 %. For small and medium-sized enterprises including start-ups, Article 99(6) caps each fine at whichever of the percentage or the amount is lower, reversing the normal rule.

Trap Applying the 7 % ceiling to an ordinary provider or deployer obligation failure; 7 % is reserved for the Article 5 prohibited practices, and the general operator tier is 3 %.

Article 53(1) gives providers of a general-purpose AI (GPAI) model four obligations regardless of any system's risk tier: keep current technical documentation of the model including its training, testing and evaluation results for the AI Office and national authorities; supply downstream providers with the information they need to understand the model's capabilities and limitations; put in place a policy to comply with Union copyright law; and publish a sufficiently detailed summary of the training content using the AI Office template. Article 53(2) exempts models released under a free and open-source licence with publicly available parameters from the first two only.

Trap Assuming an open-weight release clears all GPAI duties; the copyright policy and the public training-content summary survive the carve-out, and a model with systemic risk loses the carve-out entirely.

Article 51 classifies a general-purpose AI model as carrying systemic risk when it has high-impact capabilities judged against appropriate technical tools, benchmarks and indicators, or when the Commission decides it has equivalent capability. Article 51(2) then adds a presumption: a model is presumed to have high-impact capabilities when the cumulative computation used for its training, measured in floating point operations, is greater than 10 to the power of 25. It is a rebuttable presumption that triggers additional obligations on models already inside the GPAI category, and the Commission can adjust the threshold.

Trap Reading the compute threshold as the line that makes a model a general-purpose AI model at all; models far below it are still GPAI models with the Article 53 duty set.

Assembling and using training data is processing, so it needs its own basis under Article 6, and where the data is special-category data such as health data, biometric data used to identify a person, or trade-union membership, an Article 9 condition is required on top of the Article 6 basis rather than instead of it. Purpose is the usual point of failure: data lawfully collected to deliver a service is not automatically available as training material for a different purpose. Document the basis for the training set and the deployment separately, because a supervisory authority will ask about both.

Trap Reusing data collected for service delivery as training data on the strength of the original consent or contract basis, without re-testing the purpose.

Article 22(1) gives a person the right not to be subject to a decision based solely on automated processing, including profiling, that produces legal effects concerning them or similarly significantly affects them. Article 22(2) allows such decisions where they are necessary for a contract, authorised by Union or Member State law with suitable safeguards, or based on explicit consent, and where the contract or consent route is used Article 22(3) still requires at least the right to obtain human intervention, to express a point of view and to contest the decision. A reviewer who lacks the time, information or authority to overturn the model is the contested case, so treat a nominal human in the loop as exposure to raise with counsel rather than as a settled defence.

Trap Treating any human in the workflow as taking the decision out of Article 22 entirely, and therefore skipping the intervention, representation and contest safeguards that Article 22(3) requires even inside an exception.

Article 35(1) requires a data protection impact assessment (DPIA) before processing likely to result in a high risk to the rights and freedoms of natural persons, and it explicitly flags processing that uses new technologies. Article 35(3)(a) names systematic and extensive evaluation of personal aspects based on automated processing, including profiling, on which decisions with legal or similarly significant effects are based as a case where a DPIA is required in particular, which is exactly what a scoring or screening model does. Points (b) and (c) add large-scale special-category processing and systematic large-scale monitoring of publicly accessible areas.

Trap Scheduling the DPIA after go-live as part of an evidence pack; Article 35(1) puts it prior to the processing, so launching first is itself the violation.

Article 44 opens Chapter V of the GDPR by allowing a transfer of personal data to a third country or an international organisation only where the conditions of that chapter are met. Calling a hosted model endpoint outside the European Economic Area (EEA) sends personal data there, so the choice of endpoint carries a legal question alongside the latency and cost questions, and it needs a transfer mechanism recorded before traffic starts flowing.

Trap Treating an API call to a model hosted outside the EEA as a purely technical integration decision, so no Chapter V transfer mechanism is ever put in place.

The right to erasure under Article 17 is real but not absolute: Article 17(3) preserves processing necessary for compliance with a legal obligation, for the establishment, exercise or defence of legal claims, and for archiving, research or statistical purposes. Even where erasure does apply, removing a row from the training corpus is a different operation from removing whatever that row contributed to a set of trained parameters, and reliable technical means for the second are an open research question rather than a settled control. State plainly what your pipeline can and cannot do and let counsel judge the consequence.

Trap Promising a data subject that an erasure request removes their influence from an already-trained model, when only the stored record and future retraining are actually under your control.

ISO/IEC 42001:2023 specifies requirements for establishing, implementing, maintaining and continually improving an artificial-intelligence management system (AIMS), and was published in 2023 as the first AI management system standard. Because it is a management-system standard, an accredited certification body can certify against it, which is why buyers and tenders increasingly ask for it by name. Its clause text is paywalled, so cite its identity, scope and publication date rather than quoting requirements. ISO/IEC 23894 is the neighbouring AI risk-management guidance and is not a certifiable requirements standard.

Trap Offering NIST AI RMF adoption when a contract demands independent certification of AI governance; the framework has no certification scheme, so it cannot satisfy that clause.

The NIST AI Risk Management Framework 1.0, published as NIST AI 100-1 and released on 26 January 2023 for voluntary use, is built on four functions: GOVERN, MAP, MEASURE and MANAGE. GOVERN is the cross-cutting culture, policy and accountability layer the other three operate inside, not a first step you complete and leave behind. There is no fifth function and no certification scheme. The Generative AI Profile, NIST AI 600-1, released on 26 July 2024, is the companion profile naming risks specific to generative systems.

Trap Treating GOVERN as stage one of a four-step sequence that ends at MANAGE; it is drawn around the other functions because it applies continuously to all of them.

The EU AI Act and the GDPR apply across sectors, and neither displaces the regulator that already governed your industry. A hospital deploying a diagnostic model answers to its medical-device regime, its health-data rules and the AI Act at once; a bank deploying a credit model keeps its existing model-governance expectations while creditworthiness evaluation of natural persons also sits in the Annex III essential-services area. Expect obligations to accumulate, and answer scenario questions by listing every regime the facts trigger rather than picking the one that sounds most AI-specific.

Trap Assuming an AI Act conformity assessment discharges a sector obligation such as medical-device approval or supervisory model-risk expectations; they are separate regimes with separate evidence.

Obligations in the United States arrive from three directions at once: existing sector regulators applying existing law to AI-driven decisions, consumer-protection and anti-discrimination enforcement, and a growing patchwork of state and municipal laws imposing duties such as bias auditing or notice for automated employment decisions. Federal policy direction has shifted with successive administrations, which is why the durable common reference is the voluntary NIST AI RMF rather than a statute. The structural contrast worth carrying is that the EU regulates the technology horizontally through one binding regulation with tiers and roles, while the United States regulates the use through existing regulators plus state law.

Trap Concluding that no US federal AI statute means no US obligation; sector regulators, consumer-protection enforcement and state law reach the deployment regardless.

The Act applies in stages rather than all at once: the Article 5 prohibitions and the AI literacy duty applied from 2 February 2025, and the general-purpose AI model, governance and penalty provisions from 2 August 2025, with 2 August 2026 as the general application date in Article 113. The high-risk calendar has since been amended by the Digital Omnibus on AI, approved by the European Parliament on 16 June 2026, which sets 2 December 2027 for stand-alone systems in the Annex III areas and 2 August 2028 for high-risk AI integrated into regulated products. That amending regulation was still awaiting publication in the Official Journal at the time of writing, so treat those dates as adopted rather than consolidated. The tiers, the roles and the substance of the obligations have been stable since adoption while the dates have not, so re-check the current consolidated text before a date goes into a contract or a board paper.

Trap Quoting 2 August 2026 as the date high-risk obligations bite; that is the Act's general application date, and the Annex III high-risk duties were deferred to 2 December 2027.