Domain 2 of 4 · Chapter 1 of 3

AI Security Controls

Access control over models, datasets, and endpoints

A data scientist's notebook credential can read the raw customer table, write to the model registry, and call the production inference endpoint. One phished laptop now reaches three of the four assets this page protects, and nothing in the request path ever asked a second question. That is the shape of most AI access-control failures: not a missing firewall, but one identity doing four jobs.

Two sibling pages sit either side of this one. Where the model runs, meaning network placement, tenant isolation, and the pipeline that builds it, belongs to secure AI deployment environments; the attack taxonomy these controls reduce exposure to, meaning evasion, poisoning, extraction, inversion, and prompt injection as an attack class, belongs to adversarial AI risk mitigation. This page owns who and what may touch the training data, the model artifact, the inference endpoint, and the prompt and response record, and what proves each one is intact.

Those four assets are the spine of everything below. One section belongs to each of them, in order: data governance for the training data, integrity and provenance for the model artifact, guardrails and abuse controls for the inference endpoint, and logging and audit for the prompt and response record. Two families cut across all four instead of sitting under one, exactly as the overview's grid shows: identity, which this section covers, and encryption keys, which has its own section further down. A final pair of sections then names the framework that owns each family and the question shapes they appear in.

Four gates on one inference request

Read a single request from the outside in and you pass four authorization decisions, each with a different subject and a different owner. The figure below traces them in order.

Gate 1, the caller. Authenticate the human or the calling service and authorize them per user, not per application. A single shared API key in front of a chatbot collapses every downstream decision into one, and it is what makes the third gate impossible to enforce later.

Gate 2, the application's workload identity. The application calls the inference endpoint under its own machine identity, separate from any human's. That identity needs permission to invoke the endpoint and nothing else: not to write the registry, not to read the training bucket. Registry writers and registry readers are separate roles for the same reason, and the identity that approves a model promotion should not be the identity that can push weights, which is ordinary segregation of duties applied to a new artifact.

Gate 3, retrieval. When the application retrieves documents to ground an answer, the retrieval layer must apply the calling user's own permissions at query time. An index built by a privileged crawler and queried under the application's identity will happily return a document the asking user may not see. OWASP frames the general form of this as complete mediation: enforce the authorization check in the downstream system, not only in the layer that decided to ask.

Gate 4, the model's tools. When a model can call functions, its tool credential is an identity of its own and must be narrower than the operator's. OWASP tracks the failure as LLM06:2025 Excessive Agency[1] and splits it into three root causes worth naming separately, because each has a different fix: excessive functionality (an extension that summarises mail should not also be able to send or delete it), excessive permissions (a database credential with UPDATE and DELETE when the task needs SELECT), and excessive autonomy (a high-impact action taken with no human in the loop). The corresponding fixes are granular purpose-built extensions instead of open-ended ones such as a shell tool, minimum permissions on every downstream system, executing extensions in the individual user's context rather than under a generic privileged account, and human approval for consequential actions.

Role-based and attribute-based control, applied to AI assets

Both classic models still apply, and the exam expects the mapping rather than a new taxonomy. Role-based access control (RBAC) fits the coarse split that most AI platforms already expose: dataset reader, training-job runner, registry writer, registry reader, endpoint invoker. Attribute-based access control (ABAC) fits the finer conditions that AI work keeps needing: this dataset only for models tagged for the same purpose, this endpoint only from the production environment, this tool only during a change window. Use RBAC for the job families and ABAC for the conditions on top; that is the same layering taught for any resource, with AI assets as the objects.

A system prompt is not an access control

This is the misread worth killing on first contact. Instructions such as "never reveal the connection string" or "only answer questions about billing" sit inside the same text channel an attacker can influence, so they are guidance to a probabilistic model, not a policy engine. OWASP is explicit under LLM07:2025 System Prompt Leakage[2]: avoid using system prompts to control behaviour where possible, rely on systems outside the model, and enforce security controls independently of the model, especially privilege separation and authorization checks. Section four covers what a guardrail can and cannot do; the authorization decision belongs here, in code the model cannot talk its way past.

The takeaway is a habit for reading questions: when a stem describes an AI system, find the subject of the sentence. Whichever of the four gates that subject sits behind is the control family the answer lives in.

End user or calling service Gate 1: caller authenticate and authorize per user AI application Gate 2: workload identity invoke the endpoint, nothing else Inference endpoint retrieval tool call Gate 3: retrieval check the caller's own document permissions Gate 4: tool credential narrower scope, human approval
Four identity boundaries on one inference request: caller, application, retrieval layer, and the model's tool credential are authorized separately.

Training-data governance: provenance and minimisation

You cannot un-train a model, so every meaningful training-data control is a control you apply before the job starts. A model that memorised a customer's national identity number does not have a row you can delete; the number is spread across weights, and the only reliable remedies are retraining from a clean corpus or blocking the value on the way out. That asymmetry is why data governance is front-loaded here while the rest of this page can be layered on later.

Provenance first, because everything else is a claim about it

Record, per source dataset, where it came from, who owns it, the licence or contract that permits the use, the collection date, and the purpose it was gathered for. That record is the dataset manifest, and it is the artifact an auditor, a regulator, or an incident responder will ask for. OWASP's supply-chain guidance under LLM03:2025 Supply Chain[3] puts vetting data sources alongside vetting models and packages, for the same reason: a corpus you cannot trace is a component you cannot vouch for. The intake gate in the figure below is deliberately blunt, because a dataset admitted without a manifest is one you can never retrofit provenance onto.

Data poisoning, meaning an attacker deliberately seeding a corpus to change model behaviour, is the attack this provenance record defends against; the technique itself is taught under adversarial AI risk mitigation, and the lawful-basis question that sits behind the licence field belongs to AI regulatory frameworks.

Minimise, then choose a de-identification technique

There are five approaches worth knowing before you reach for any one of them, and the selection criterion is what the training task actually needs from the field.

Approach What it does Reach for it when
Field removal (minimisation) The column never enters the corpus The task does not need the field at all, which is more often than teams assume
Pseudonymisation Direct identifiers replaced by a consistent surrogate You need per-subject grouping across rows but not the real identity
Tokenisation Value replaced by a token, with the original held in a separate vault and recoverable by lookup A downstream process must be able to resolve the original under separate authorization
Aggregation Individual rows replaced by counts or statistics The signal you need is a distribution, not a person
Differential privacy Calibrated noise added so individual records cannot be reverse-engineered from the output You must publish or train on a sensitive population and can accept a measured accuracy cost

Two techniques sit outside this ladder because they change where training happens rather than what is in the record. Federated learning trains across decentralised holders so the raw data never pools in one place, and homomorphic encryption allows computation on encrypted values; OWASP lists both under LLM02:2025 Sensitive Information Disclosure[4] alongside differential privacy. Treat them as specialised, not default: each carries real accuracy, latency, or engineering cost.

Masking is not one of these. Masking systematically removes or obscures a field for display, such as showing only the last four digits of a card number, and it is a presentation-layer control. It does not protect the stored record, and it is not a training-corpus control. Tokenisation is the one that removes the raw value from the store while keeping a route back to it. Getting this pair the wrong way round is a classic distractor.

Classification travels with the data

A derived artifact inherits the sensitivity of its worst input. A retrieval corpus assembled from a document library carries the classification of the most sensitive document in it, and so does the embedding index built from that corpus, and so does the model fine-tuned on it. Label the corpus at intake and let that label drive the key boundary, the access role, and the retention rule, rather than re-deciding sensitivity at each stage.

Retention and deletion close the loop. OWASP's LLM02 guidance asks for clear data-retention and deletion policies and a route for users to opt out of having their data included in training. In practice that means the manifest records a retention period per source, and the deletion path reaches the corpus, the derived embeddings, and the logs, not just the primary database.

The takeaway: minimise before you de-identify, record provenance before you train, and remember that a control applied after the training job is a compensating control, never the primary one.

Raw source dataset Provenance recorded? no yes Classified and minimised? no yes Not admitted Approved training corpus Training job under a scoped role Model artifact with data lineage
Training-data intake gates: without a provenance record and without minimisation, a dataset never reaches the corpus the model learns from.

Model integrity and the AI supply chain

A model file is executable input, not a data file. The default checkpoint format for PyTorch weights is Python's pickle serialisation, and Hugging Face's own security documentation states plainly that there are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file[5]. The mechanism is worth knowing because it explains why scanning is weak and format choice is strong: a pickle is a stack of opcodes replayed at load time, and the GLOBAL and STACK_GLOBAL opcodes import Python objects while REDUCE calls one with attacker-supplied arguments. Importing exec and reducing it over a string is enough. Nothing has to be "run" in the ordinary sense, because deserialising the file is the execution. Loaders have since added restricted unpickling modes that admit only tensor data, but whether one is active depends on the library version and on how the load is called, which is why the format decision and the digest check are the controls that do not rest on a default.

So the first control is a format decision, not a scanner. The safetensors format exists precisely to be the boring alternative, described by its maintainers as a simple format for storing tensors safely, as opposed to pickle[6]: it holds tensor data and a header, and has no mechanism to execute code on load. Prefer it for anything you did not produce yourself. Scanning still has a place as a second layer, and it is genuinely partial: Hugging Face runs a pickle-import scan that reads the opcodes with pickletools without executing them and flags suspicious imports, and its own documentation calls that best-effort rather than foolproof.

The promotion gate

Treat an external model the way you treat any other third-party binary, with the gates in the figure below applied in order before it can reach production.

  1. Land it somewhere isolated. Download into a staging area with no production credentials and no path to the registry, so a load-time payload has nothing worth reaching.
  2. Verify the digest and the signature. OWASP's guidance is direct: only use models from verifiable sources, and use third-party model integrity checks with signing and file hashes (a file hash is the digest) to compensate for the lack of strong model provenance. Record the digest; the point of recording it is that you can re-check it at load time in production, not only at download time.
  3. Require a non-executing weights format, or convert to one before promotion.
  4. Inventory it. OWASP asks for an up-to-date, accurate, signed component inventory in the form of a software bill of materials (SBOM), and points at OWASP CycloneDX as the emerging bill-of-materials format for AI components. Alongside it, the model card records the base model, training data description, intended use, and known limitations. The inventory entry is what makes the next vulnerability disclosure answerable in minutes instead of days.
  5. Only then register and deploy, with the registry entry signed and the digest re-checked when the serving process loads the file.

The supply chain is wider than the weights file. Adapters and low-rank fine-tuning layers merged onto a base model, quantised re-uploads of a well-known model, the inference server and its Python dependencies, and the datasets from the previous section are all components with the same intake question. Naming only the weights file is the common gap.

Where the standards put this

Two documents own this ground and both are fair game as answer options. NIST SP 800-218A[7], Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile, published July 2024, augments the Secure Software Development Framework with AI-specific practice, and is explicitly aimed at producers of AI models, producers of systems that use those models, and acquirers of those systems. That last audience is the one most candidates forget: buying a model puts you in scope. The joint Guidelines for Secure AI System Development[8], published November 2023 by the UK National Cyber Security Centre with CISA and international partners, splits the lifecycle into secure design, secure development, secure deployment, and secure operation and maintenance; supply-chain security and asset documentation sit in the secure development stage.

The build pipeline that produces your own models, including its runners, artifact stores, and promotion automation, is hardened under secure AI deployment environments. This section stops at the artifact and its provenance.

The takeaway: a model you did not build is untrusted code with a licence attached. Digest, signature, format, scan, inventory, in that order, and re-check the digest where it is loaded.

Third-party model artifact Isolated staging area Digest and signature verified? no yes Do not promote Non-executing weights format? no yes Convert first Scanned and inventoried? no yes Hold for review Signed model registry entry Production endpoint digest re-checked at load
Promotion gates for a third-party model: verify digest and signature, require a non-executing format, then inventory it before the registry accepts it.

Guardrails on prompts and completions

Teams build the input filter and stop. The breach then arrives through the completion: a support agent's browser renders a model-generated answer containing a script tag, or a helper pastes a model-generated query straight into a database call. Because a completion is shaped by the prompt, and the prompt is attacker-influenced, the model's output is attacker-influenced too. Both directions of the endpoint need a checkpoint, and the figure below shows where each one sits.

Three places a guardrail can live

Decide the placement before the policy, because it determines what the guardrail can see and who can bypass it.

  • In the application, around the model call. Maximum control, and it travels with your code, but every application team re-implements it and drift is invisible.
  • As a separate moderation service the application calls. One policy shared across applications; the application must actually call it, so the bypass risk is an application-code review problem.
  • Bound to the model invocation by the platform. The narrowest bypass surface, at the cost of coupling to that platform's policy vocabulary.

Two managed examples, among others: Amazon Bedrock Guardrails[9] can be attached to the inference call by guardrail identifier and version, or invoked on its own through an ApplyGuardrail call, and Azure AI Content Safety[10] exposes its detections as standalone APIs. Naming a specific product is never the point of an exam item; recognising the policy type behind it is.

The policy types worth recognising by name

Across implementations the same families recur, and Bedrock Guardrails is a convenient enumeration of them because its documentation names each one: content filters over harm categories such as hate, insults, sexual, violence, misconduct, and prompt attack, with a configurable strength per category; denied topics defined for your application's context; word filters for exact-match blocklists including profanity and competitor names; sensitive information filters that block or mask personally identifiable information (PII) with regular-expression patterns as an extension; and contextual grounding checks that flag a response not grounded in the retrieved source. Azure AI Content Safety covers the same ground under different names, including Prompt Shields for user-input attacks against a model and groundedness detection for retrieval-augmented answers.

Two properties matter more than the catalogue. First, these evaluate both the input prompt and the model completion; a guardrail configured on one side only is half a control. Second, the grounding check is the one that addresses answers invented from nothing, and it needs the retrieved source passed alongside the response, so it exists only in retrieval-augmented designs.

The output side is an injection problem, not a content problem

Content filtering and output handling are different controls with different failure modes, and conflating them is the expensive mistake. OWASP separates them: LLM05:2025 Improper Output Handling[11] covers insufficient validation and sanitisation of model output before it reaches another component, and its named consequences are ordinary application vulnerabilities: cross-site scripting when unfiltered output is rendered in a browser, SQL injection when a generated query runs unparameterised, path traversal from a constructed file path, and remote code execution when output reaches a shell or an eval. Its first prevention strategy is the sentence to remember: treat the model as any other user, adopt a zero-trust approach, and apply proper input validation on the model's responses.

Concretely, that means the deterministic layer in the figure: context-aware output encoding chosen for the sink (HTML, JavaScript, SQL, or shell), parameterised statements for anything that touches a database, a Content Security Policy on pages that render model output, and schema validation where the model was asked for structured output. OWASP's prompt-injection guidance under LLM01:2025 Prompt Injection[12] reinforces the same split: define expected output formats and validate adherence with deterministic code, and handle privileged functions in application code with the application's own tokens rather than handing them to the model.

A guardrail is probabilistic; validation is deterministic. A content filter returns a judgement with a configurable strength and will sometimes be wrong in both directions. A schema check, an allowlist, and a parameterised query either hold or fail loudly. Use the guardrail for what only a model can judge, meaning tone, topic, and harm, and use deterministic validation for anything with a definable correct shape. Neither one is an authorization control, which is the point made in the first section and worth carrying here: a filter decides whether text is acceptable, never whether this caller may have it.

Two habits that finish the control

Pin the guardrail policy version to the deployed endpoint so a policy change is a reviewable deployment rather than a silent console edit, and decide the failure mode explicitly: if the guardrail service is unavailable, an endpoint that fails open has no guardrail exactly when it is under load.

The takeaway: filter in, filter out, then encode and validate before the completion touches anything that renders, queries, or executes.

User prompt Input guardrail harm categories, denied topics, PII, prompt attack blocked Request blocked allowed Model inference Output guardrail content policy, PII masking, grounding blocked Response blocked allowed Deterministic output handling encode, parameterise, validate schema Downstream sink browser, database, shell, tool call
Two guardrail checkpoints plus a deterministic layer: the prompt is filtered in, the completion filtered out, then encoded before it reaches any sink.

Abuse controls on the inference endpoint

An inference endpoint bills per token, so an endpoint without quotas is a budget with an API in front of it. That single sentence covers this section's scope: the controls that bound how much an authenticated or anonymous caller can consume, and therefore how much an attacker can extract or spend on your behalf.

OWASP collects these under LLM10:2025 Unbounded Consumption[13], which is worth reading as three distinct harms rather than one:

  • Denial of wallet. The attacker's goal is your invoice. High volumes of ordinary-looking operations against a pay-per-use service drain the budget without ever breaching anything. Availability controls sized for traffic will not catch it, because the traffic is affordable to send and expensive to serve.
  • Model extraction. Careful querying at volume harvests enough input-output pairs to train a functional equivalent of a proprietary model. The economics are the control surface: extraction needs many queries, so a per-identity quota raises the attacker's cost far more than a global endpoint limit, which they can hide inside.
  • Resource depletion. Variable-length or deliberately expensive inputs exhaust compute and degrade service for everyone else.

The control set

The mitigations OWASP lists group into four layers, and a well-written question usually rewards the layer that matches the harm in the stem.

Bound the input. Strict validation and explicit size limits on prompt length, uploaded file size, and any retrieval expansion, applied before the request reaches a model. This is the cheapest control and the one most often missing.

Bound the caller. Rate limiting and quotas per authenticated identity, which is why gate one in the first section had to authenticate per user rather than behind a shared key. Per-IP limits alone are trivially spread across a botnet and punish shared corporate egress addresses.

Bound the request. Timeouts, throttling, and graceful degradation so a single expensive generation cannot hold capacity indefinitely, and so the service sheds load predictably instead of failing at the worst moment.

Watch the pattern. Comprehensive logging of consumption alongside anomaly detection on query patterns, because extraction looks like enthusiastic use until you plot it: systematic coverage of the input space, unusual entropy in prompts, or a flat request rate that never sleeps. Budget alarms belong here too, as a detective control that catches the harm the preventive limits missed.

OWASP additionally lists model-side measures against extraction, including watermarking frameworks and adversarial robustness training. Those protect the model's value rather than the endpoint's capacity, and the attack techniques behind them are taught under adversarial AI risk mitigation; the detection side of spotting the campaign in telemetry belongs to AI-driven detection and response.

The takeaway: bound the input, the caller, and the request before you bound the bill. If the stem mentions cost, volume, or a suspicious query pattern against an endpoint, the answer is a quota or a limit tied to an identity, not more capacity.

Encryption, key boundaries, and secrets

Give each of the four assets its own key boundary, and revoking one key revokes exactly one asset. That is the whole design goal, and it is why "we encrypt everything with the platform default key" is a weaker answer than it sounds: one key that covers the training corpus, the registry, and the prompt and response record cannot be revoked for any of them without breaking the other two.

Where the ciphertext actually has to be

An AI pipeline has more storage locations than a conventional application, and each is a place a control can be forgotten:

  • the training corpus and its staging copies;
  • model checkpoints during training, which are often written to a scratch location nobody classified;
  • the model registry and the artifact store behind it;
  • the vector or feature store built from the corpus;
  • the prompt and response record, covered in the next section.

Encrypt each at rest under a key you can name, and encrypt in transit on every hop including the internal call from the application to the model host. Customer-managed keys, sometimes offered as bring-your-own-key, are what turn encryption from a checkbox into a control: Microsoft's documentation for Azure AI Content Safety describes them as giving the flexibility to create, rotate, disable, and revoke access controls, and to audit the encryption keys used to protect your data[10]. Rotation, revocation, and the key-use audit trail are the properties an exam item is testing when it contrasts a platform-managed key with a customer-managed one.

Data in use is the third state, and the honest answer is that it is specialised. Homomorphic encryption, which OWASP names among the privacy techniques under LLM02, and confidential-computing enclaves both allow processing without exposing plaintext to the host, at a real cost in performance and engineering effort. Reach for them when the threat model genuinely includes the operator of the compute, not as a default.

An embedding is not anonymisation

A vector store built from sensitive documents inherits their classification. Embeddings are lossy, which invites the assumption that they are safe to store loosely, but they are derived from the source text and retrieval returns the source chunk anyway. Key, classify, and access-control the vector store exactly as you would the documents it was built from, and apply the retrieval-time authorization check from the first section on top.

Secrets belong outside the prompt

The rule has one line: nothing the model can read should be a secret. OWASP is unambiguous under LLM07:2025 System Prompt Leakage[2], which asks you to avoid embedding sensitive information such as API keys, authentication keys, database names, user roles, and permission structures in prompts, and to externalise that data to systems the model cannot access. A system prompt is not a vault, and it is not private: it is text a determined prompt will eventually surface.

In practice: the application fetches credentials from a secret store at call time under the workload identity from the first section, tool credentials are issued to the tool layer rather than described to the model, and connection strings never appear in a prompt template. The same rule reaches backwards into training: credentials and keys committed to a code corpus become memorised strings a completion can reproduce, so secret scanning belongs in the data-intake gate of the second section as well as in the repository.

The takeaway: one key boundary per asset, customer-managed where you need revocation and an audit trail, and no secret anywhere the model can read.

Logging and audit of prompts and responses

Three months after a deployment, someone asks what the assistant told a specific customer on a specific day and under which policy. If the answer is "we log request counts", there is no incident response, no evidence for a regulator, and no way to tell a model defect from an attack. The prompt and response record, which lives in the inference log, is the fourth asset for exactly this reason, and it is the one teams most often treat as debug output.

What an inference log entry has to carry

Field by field, the entry should let a reviewer reconstruct the decision without access to the running system:

Field Why it earns its place
Request identifier and timestamp Correlates the entry with application and platform logs
Caller identity The per-user identity from gate one, not the shared application account
Model identifier and version A behaviour change after a model swap is otherwise unattributable
Guardrail policy version and decision Distinguishes "the filter allowed it" from "no filter ran"
Tool calls attempted, approved, and refused The audit trail for the agency controls in the first section
Token or unit counts Feeds the consumption controls in the previous section
Prompt and completion content The part with the evidentiary value, and the part with the sensitivity

The last row is the hard one. Content is what makes the log useful and what makes it dangerous, because the log store then holds whatever a user typed, including data they should never have pasted. Classify the store at the level of the most sensitive content it can receive, and apply the sensitive-information filter from the guardrail section at write time so identifiers are masked before they land. Where the filter is probabilistic, treat the store as if it failed.

Integrity, access, and retention

Integrity. Write to an append-only store the application identity cannot rewrite or delete. If the identity that produces the log can also edit it, a compromise of the application is a compromise of the evidence.

Access. Reading prompt content is a separate, narrower permission than writing it, granted to named security and privacy reviewers, and every read is itself logged. Emergency access follows a break-glass path that is time-boxed and alerts on use, because a log store containing customer conversations is a more attractive target than the endpoint that generated them.

Retention. Keep the shortest period that satisfies the obligation, set per source in the same manifest that governs training data, and make sure the deletion path actually reaches the logs. A deletion request honoured in the primary database but not in the inference log is not honoured. OWASP's LLM02 guidance asks for exactly this pairing of clear retention and deletion policies with user control over inclusion.

These records have a second life as detection telemetry: guardrail decisions, refused tool calls, and anomalous prompt patterns are signal a security operations centre can hunt over. Building that detection, tuning it, and running the analyst workflow is the subject of AI-driven detection and response; this page stops at producing a record that is complete, tamper-evident, and lawfully retained.

The takeaway: log the decision, not just the call. Identity, model version, guardrail verdict, tool outcome, and content, in a store the application cannot rewrite and a reviewer cannot read casually.

Which framework owns which control

SecAI+ answer options are usually written in a published framework's vocabulary rather than in generic security language, so knowing which body owns a control is often the whole question. Seven artifacts cover almost everything on this page, and each has a distinct job.

NIST AI Risk Management Framework 1.0, published as NIST AI 100-1 in January 2023[14], is the voluntary risk framework. Its core has four functions: GOVERN, MAP, MEASURE, and MANAGE. GOVERN is the cross-cutting function that runs through the other three rather than a first stage you complete and leave behind, and MAP, MEASURE, MANAGE is the working order for a given system: establish context and identify risks, analyse and track them, then act on them. Controls on this page are mostly MANAGE actions justified by MAP and MEASURE findings, under policies GOVERN set.

NIST AI 600-1, the Generative AI Profile[15] published July 2024, is a cross-sectoral companion profile to AI RMF 1.0 that tailors it to generative systems. It is a profile, not a replacement, which is the distinction a question about "which document extends the AI RMF for generative AI" is testing.

NIST SP 800-218A is the secure-development answer, covered in the model-integrity section above: an SSDF community profile for generative AI and dual-use foundation models, addressed to model producers, system producers, and acquirers.

The OWASP Top 10 for LLM Applications is the application-layer catalogue, and it is the one whose identifiers appear verbatim in answer options. The 2025 edition[16] reads: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption. Entries move between editions, so state the edition whenever you quote an identifier; the ones on this page are all 2025.

MITRE ATLAS is the adversary-technique knowledge base for AI systems, structured like ATT&CK with tactic and technique identifiers. It describes what an attacker does rather than what you should implement, so it belongs to the attack pages, not to a controls answer.

ISO/IEC 42001 is the management-system answer. Its standard page[17] describes it as specifying requirements for establishing, implementing, maintaining, and continually improving an artificial intelligence management system, applicable to any organisation that provides or uses products or services using AI systems. Being a management system standard is the point: it is certifiable and it governs the organisation's process, where the AI RMF is a voluntary framework and neither is a law. The clause text sits behind a paywall, so treat the scope statement as the citable claim.

The joint Guidelines for Secure AI System Development, also covered above, organise the lifecycle into secure design, secure development, secure deployment, and secure operation and maintenance.

Mapping the control families

Read the table below as a zoom-in on this page's own sections rather than a rival taxonomy: one row per section, in section order, with a closing row for the organisational governance that sits above all of them and belongs to AI lifecycle GRC integration.

Control family on this page Primary owning artifact Secondary tie-in
Identity and access over assets NIST AI RMF MANAGE OWASP LLM06 for the model's tool permissions
Training-data provenance and minimisation NIST AI RMF MAP and MANAGE OWASP LLM02 and LLM03
Model integrity and supply chain NIST SP 800-218A OWASP LLM03; Guidelines, secure development stage
Guardrails on prompts and completions OWASP LLM01 and LLM05 NIST AI 600-1 for generative-specific risk
Abuse and consumption controls OWASP LLM10 Guidelines, secure operation and maintenance
Encryption, keys, and secrets NIST AI RMF MANAGE OWASP LLM07 for secrets in prompts
Logging and audit of interactions Guidelines, secure operation and maintenance ISO/IEC 42001 for the governing process
Organisational governance around all of it ISO/IEC 42001 NIST AI RMF GOVERN

Regulatory obligations, meaning the EU AI Act risk tiers and data-protection duties, are a separate question type covered under AI regulatory frameworks, and running these controls as a programme across the AI lifecycle is the subject of AI lifecycle GRC integration.

The takeaway: framework, profile, management system, technique catalogue, and guidance are five different kinds of document. Match the kind the question asks for before you match the topic.

Exam patterns: reading a controls question

Controls questions on this exam share a shape: a two-sentence scenario, one asset under pressure, and four options that are all real controls. Three of them protect a different asset or a different layer, which is why naming the asset before reading the options is worth the two seconds it costs. Domain 2 carries 40% of a maximum of 60 questions in 60 minutes[18], some of them performance-based, so this pattern recognition is the highest-value habit on the page.

"A team downloads a popular model from a public hub and loads it on a training server." The asset is the model artifact and the exposure is load-time code execution. The right answer verifies integrity and provenance before the file is loaded: check the publisher signature and the recorded digest, and prefer a non-executing weights format. Antivirus scanning is the tempting distractor because it sounds like the same job, but signature-based scanning does not understand pickle opcodes; treat it as a partial second layer. Network isolation of the training server is a real control that belongs to the deployment-environment page and does not stop the payload from running with that server's credentials.

"A model is fine-tuned on support tickets, and a completion later returns a customer's full postal address." The asset is the training data and the failure is memorisation, so the correct control is applied before training: minimise the fields, then de-identify the ones that remain. Encrypting the model at rest is the classic wrong answer, because encryption protects the artifact from someone who steals the file, not from the model reproducing what it learned. An output filter that masks identifiers is a legitimate compensating control and can appear as a partially correct option; if both appear, the pre-training control is the primary one.

"An assistant's answer is rendered in an internal support portal, and a crafted ticket causes a script to run in an agent's browser." The asset is the inference endpoint's output path, the identifier is LLM05 Improper Output Handling, and the answer is context-aware output encoding before rendering, backed by a Content Security Policy. Strengthening the system prompt is the distractor that catches candidates who read this as a prompt problem; the model is behaving as instructed by an attacker, and the fix is downstream of it.

"Inference costs quadrupled overnight with no rise in registered users." The asset is the inference endpoint and the harm is denial of wallet under LLM10 Unbounded Consumption. The answer is rate limiting and quotas tied to an authenticated identity, plus input size limits. Adding capacity or enabling autoscaling is the trap: it removes the symptom and increases the bill. A per-IP limit alone is the weaker of two similar options, because it spreads across sources and penalises shared egress addresses.

"A scheduling assistant with mailbox access deleted a thread while summarising it." The asset is the model's tool credential, the identifier is LLM06 Excessive Agency, and the root cause is excessive functionality: the extension carried delete capability the task never needed. The answer narrows the extension to read-only and adds human approval for consequential actions. Instructing the model not to delete anything is the distractor, and it fails for the reason established in the first section: a prompt is not a permission.

"An organisation wants to demonstrate to customers that its AI practices are managed and auditable." This is a document-kind question. ISO/IEC 42001 is the certifiable management system standard; the NIST AI RMF is a voluntary framework with no certification; the EU AI Act is a regulation, so it imposes obligations rather than offering a demonstration; and the OWASP list is an application-risk catalogue. Match the kind of artifact before the topic.

A final habit for performance-based items: these frequently ask you to place controls against assets or against framework functions. Rehearse the four-asset split and the mapping table above until you can produce them cold, because a drag-and-drop item gives no partial credit for knowing the control without knowing where it attaches.

The four AI assets and the control family that protects each

Control questionTraining dataModel artifactInference endpointPrompt and response record
Primary control familyData governance: provenance, minimisation, classificationIntegrity and provenance: digest, signature, non-executable formatAuthorization and guardrails: scoped identity, content filters, quotasAudit and retention: what is recorded, for how long, who may read it
Who may reach itData engineers and the training job role, not the whole ML teamRegistry writers are a separate role from registry readersCallers authenticated per user; the model's tool role is scoped separatelySecurity and privacy reviewers, under logged emergency (break-glass) access
What proves it is intactA dataset manifest naming source, licence, and collection date per setA recorded digest plus a signature checked at load timeA guardrail policy version pinned to the deployed endpointAn append-only log store the application identity cannot rewrite
Main OWASP LLM Top 10 (2025) tie-inLLM02:2025 Sensitive Information DisclosureLLM03:2025 Supply ChainLLM06:2025 Excessive Agency and LLM10:2025 Unbounded ConsumptionLLM02:2025 again, on the stored-log side
What failure looks like without itMemorised personal data reproduced verbatim in a completionBackdoored weights, or code execution the moment the file is loadedDenial of wallet, cheap model extraction, unauthorised tool actionsNo defensible answer to what the system said, to whom, and when

Decision tree

Training data or retrieval corpus? yes Data governance provenance, minimisation, classification no Model artifact or registry entry? yes Integrity and provenance digest, signature, non-executable format no Inference endpoint or its tools? yes Authorization and guardrails scoped identity, content filters, quotas no Audit and retention what is recorded, how long, who may read Always: least privilege and one key boundary per asset

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Name the exposed asset before you choose a control

Every control on an AI system attaches to one of four assets: the training data, the model artifact, the inference endpoint, or the prompt and response record. Reading a scenario for the asset first collapses four plausible options to one family, because data governance protects the corpus, integrity and provenance protect the weights file and its registry entry, authorization and guardrails protect the endpoint, and audit and retention protect the interaction record. Least privilege and key management are the two families that cut across all four rather than sitting under one.

A system prompt is guidance to a model, never an access control

Instructions such as "never reveal the connection string" sit in the same text channel an attacker can influence, so they steer a probabilistic model rather than enforce a policy. OWASP's LLM07:2025 System Prompt Leakage guidance says to avoid using system prompts to control behaviour where possible and to enforce security controls, especially privilege separation and authorization checks, in systems outside the model. The authorization decision belongs in application code the model cannot talk its way past.

Trap Hardening the system prompt in answer to a data-disclosure scenario; the prompt is recoverable text, and the control it was standing in for is an authorization check.

Give the model's tools their own credential, narrower than the operator's

When a model can call functions, that tool credential is a separate identity and carries only the privilege the task needs. OWASP splits the failure, LLM06:2025 Excessive Agency, into three root causes with three different fixes: excessive functionality, where an extension that summarises mail can also send and delete it, fixed by a granular purpose-built extension; excessive permissions, where a database credential holds UPDATE and DELETE for a SELECT task, fixed by minimum permissions in the downstream system; and excessive autonomy, where a high-impact action runs unreviewed, fixed by human approval. Executing extensions in the individual user's context rather than under a generic privileged account keeps those fixes from being undone by a shared account.

Trap Reaching for an open-ended tool such as a shell wrapper because it covers future needs; it hands the model every capability the host has instead of the one the task requires.

Apply the caller's own document permissions at retrieval time

A retrieval index assembled by a privileged crawler returns any document it holds unless the query itself is filtered by the asking user's permissions, so the check belongs in the retrieval layer at query time. OWASP frames the general rule as complete mediation, meaning the downstream system performs its own authorization check rather than trusting the layer that decided to ask. Skip it and one shared index makes every user a reader of every other user's documents.

Trap Relying on the application's front door to authorize the user and treating the index as internal; the front door authorized a session, not a document.

You cannot un-train a model, so data controls run before the job

A value a model memorised is spread across weights rather than stored in a row you can delete, so the only remedies afterwards are retraining from a clean corpus or blocking the value on the way out. That makes minimisation, de-identification, and provenance pre-training controls, and everything applied later a compensating control. OWASP tracks the resulting exposure as LLM02:2025 Sensitive Information Disclosure, where a completion reproduces personal data the corpus contained.

Trap Answering a memorised-personal-data scenario with encryption of the model at rest; encryption stops someone who steals the file, not the model reproducing what it learned.

Record provenance per source dataset before it enters the corpus

The dataset manifest names the source, the owner, the licence or contract permitting the use, the collection date, and the purpose the data was gathered for, one entry per source dataset. It is the artifact an auditor or an incident responder asks for, and it cannot be reconstructed after the fact, which is why an unrecorded dataset is rejected at intake rather than fixed later. OWASP's LLM03:2025 Supply Chain guidance puts vetting data sources alongside vetting models and packages, on the same reasoning: a corpus you cannot trace is a component you cannot vouch for.

Remove the field before you try to disguise it

Data minimisation, meaning the column never enters the corpus at all, is the strongest and cheapest de-identification approach and applies more often than teams assume, because a training task rarely needs the identifier itself. Pseudonymisation keeps per-subject grouping without the real identity, aggregation replaces rows with counts or statistics when the signal you need is a distribution, and differential privacy adds calibrated noise so an individual record cannot be reverse-engineered from the output, at a measured accuracy cost. Choose by what the task actually needs from the field, then apply the weakest technique that still satisfies it.

Masking hides a value on screen; tokenisation removes it from the store

Masking systematically removes or obscures a field for display, such as showing only the last four digits of a card number, so it is a presentation-layer control and the stored record still holds the real value. Tokenisation replaces the value with a surrogate and keeps the original in a separate vault, recoverable by lookup under separate authorization, so it is the technique that actually takes the raw value out of the store. Both are legitimate controls; only one of them changes what a training corpus or a log file contains.

Trap Choosing masking to protect data at rest; masking is applied at presentation time and leaves the underlying record intact.

Loading a pickle checkpoint runs code, so prefer safetensors

The default PyTorch checkpoint format is Python pickle, and Hugging Face's security documentation states that dangerous arbitrary code execution attacks can be perpetrated when you load a pickle file. The mechanism is the opcode stream replayed at load time: GLOBAL and STACK_GLOBAL import a Python object and REDUCE calls it with attacker-supplied arguments, so importing exec is enough to run anything. The safetensors format stores tensor data and a header with no mechanism to execute code on load, which makes the format choice a stronger control than any scan or loader default.

Trap Treating an antivirus scan as sufficient before loading third-party weights; signature scanning does not interpret pickle opcodes, and the pickle-import scanners that do are documented as best-effort.

Verify a third-party model by signature and digest, then re-check at load

OWASP's LLM03:2025 Supply Chain guidance is direct: use models only from verifiable sources, and use third-party model integrity checks with signing and file hashes, meaning the digest, to compensate for the lack of strong model provenance. Record the digest at intake so the serving process can re-check it when it loads the file, because a digest verified only at download proves nothing about the artifact sitting in the registry a month later. Do the verification in an isolated staging area holding no production credentials, so a load-time payload has nothing worth reaching.

Every model needs an inventory entry and a model card

OWASP asks for an up-to-date, accurate, signed component inventory in the form of a software bill of materials (SBOM), and points at OWASP CycloneDX as the emerging bill-of-materials format for AI components. The model card sits beside it and records the base model, a description of the training data, the intended use, and the known limitations. The pair turns the next vulnerability disclosure into a query instead of an investigation, and its scope includes fine-tuning adapters, quantised re-uploads, and the inference server's dependencies, not only the weights file.

NIST SP 800-218A binds the buyer of a model, not just its builder

SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models, is a community profile of the Secure Software Development Framework published in July 2024, and it names three audiences: producers of AI models, producers of the systems that use those models, and acquirers of those systems. Downloading or purchasing a model therefore brings an organisation into scope for the profile's practices rather than exempting it.

Trap Assuming the generative-AI secure-development profile applies only to organisations that train their own models; the acquirer is named as an audience explicitly.

A guardrail configured on one direction is half a control

Managed guardrail services evaluate both the input prompt and the model completion, because the two directions catch different failures. What arrives is attacker-influenced text needing content filtering, topic denial, and size limits; what leaves is shaped by that same text, so it needs its own pass before anything downstream sees it. Decide the failure mode explicitly as well: a guardrail that fails open leaves the endpoint unprotected exactly when it is under load.

Recognise the guardrail policy families by what each one decides

The same families recur across implementations: content filters that score harm categories such as hate, insults, sexual, violence, misconduct, and prompt attack with a configurable strength per category; denied topics defined for the application's own context; word filters holding exact-match blocklists; sensitive information filters that block or mask personally identifiable information (PII), with regular-expression patterns as an extension; and contextual grounding checks that flag a response not grounded in the retrieved source. Amazon Bedrock Guardrails and Azure AI Content Safety are two implementations among others, and the grounding check exists only in retrieval-augmented designs, because it needs a retrieved source to compare the answer against.

Treat a completion as untrusted input to whatever renders or runs it

OWASP's LLM05:2025 Improper Output Handling covers insufficient validation of model output before it reaches another component, and its named consequences are ordinary application vulnerabilities: cross-site scripting when output is rendered in a browser, SQL injection when a generated query runs unparameterised, path traversal from a constructed file path, and remote code execution when output reaches a shell. Its first prevention strategy is the line to keep: treat the model as any other user, adopt a zero-trust approach, and validate its responses. Concretely that means context-aware encoding chosen for the sink, parameterised statements for anything touching a database, a Content Security Policy on pages that render completions, and schema validation where structured output was requested.

Trap Fixing a rendered-script incident by strengthening the system prompt; the model produced exactly what the attacker asked for, and the missing control sits downstream of it.

Guardrails judge; deterministic validation decides

A content filter returns a judgement at a configurable strength and will occasionally be wrong in both directions, so it belongs where only a model can judge: tone, topic, and harm. A schema check, an allowlist, and a parameterised query either hold or fail loudly, so they belong wherever the correct shape can be defined. Use both, and remember that neither is an authorization control: a filter decides whether text is acceptable, never whether this caller may have it.

Rate limit per authenticated identity, because that is what extraction costs

OWASP's LLM10:2025 Unbounded Consumption gathers three harms behind one control surface: denial of wallet, where ordinary-looking volume drains a pay-per-use budget; model extraction, where enough query and response pairs train a functional equivalent of your model; and resource depletion from deliberately expensive inputs. Extraction needs many queries, so a per-identity quota raises the attacker's cost far more than a global endpoint limit they can hide inside, and a per-IP limit alone spreads across sources while penalising shared corporate egress addresses. Pair the quota with input size limits, request timeouts, throttling, and anomaly detection over query patterns.

Trap Answering a sudden inference cost spike by adding capacity or enabling autoscaling; that removes the symptom and enlarges the bill the attacker is running up.

Give each asset its own key so revocation stays surgical

One platform-default key covering the training corpus, the model registry, and the prompt and response record cannot be revoked for any one of them without breaking the other two, which is the whole argument for a key boundary per asset. Customer-managed keys, sometimes offered as bring-your-own-key, supply the properties a question is usually testing: create, rotate, disable, and revoke, plus an audit trail of which key protected what. Cover the storage locations teams forget as well as the obvious ones, particularly training checkpoints written to scratch storage and the vector store built from the corpus.

Nothing the model can read is a secret

OWASP's LLM07:2025 System Prompt Leakage guidance asks you to avoid embedding sensitive information such as API keys, authentication keys, database names, user roles, and permission structures in prompts, and to externalise that data to systems the model cannot access. In practice the application fetches credentials from a secret store at call time under its own workload identity, tool credentials are issued to the tool layer rather than described to the model, and connection strings never appear in a prompt template. The same rule reaches backwards into training data, where a credential committed to a code corpus becomes a memorised string a completion can reproduce.

Trap Storing an API key in the system prompt on the reasoning that users never see it; the system prompt is recoverable text, which is exactly what LLM07 names.

Log the decision, not just the call

An inference log carrying only a timestamp and a token count cannot answer what the system said, to whom, and under which policy. Record the request identifier, the per-user caller identity rather than the shared application account, the model identifier and version, the guardrail policy version and its verdict, the tool calls attempted, approved, and refused, the unit counts, and the prompt and completion content. Write it to an append-only store the application identity cannot rewrite, because if the identity that produces the evidence can also edit it, compromising the application compromises the record.

A deletion honoured only in the primary database is not honoured

The deletion path has to reach every derived copy: the training corpus, the embeddings built from it, and the inference logs that captured whatever a user typed into the prompt. OWASP's LLM02 guidance pairs clear data retention and deletion policies with a route for users to opt out of having their data included in training, and the retention period is set per source in the same manifest that governs the corpus. Keep the shortest period the obligation allows, because a log store holding customer conversations is a more attractive target than the endpoint that produced them.

Pin the guardrail policy version to the deployed endpoint

A guardrail policy edited live is a security control with no change history, so bind a specific policy version to the endpoint and ship a policy change the way you ship code. Managed guardrail services are invoked by identifier and version for exactly this reason, which also lets a bad policy be rolled back without touching the model. The version then belongs in every log entry, so a later review can tell which policy was in force when a given completion was allowed.

Also tested in

References

  1. LLM06:2025 Excessive Agency Whitepaper
  2. LLM07:2025 System Prompt Leakage Whitepaper
  3. LLM03:2025 Supply Chain Whitepaper
  4. LLM02:2025 Sensitive Information Disclosure Whitepaper
  5. Pickle Scanning
  6. Safetensors
  7. NIST SP 800-218A: Secure Software Development Practices for Generative AI and Dual-Use Foundation Models Whitepaper
  8. Guidelines for Secure AI System Development Whitepaper
  9. Detect and filter harmful content by using Amazon Bedrock Guardrails
  10. What is Azure AI Content Safety?
  11. LLM05:2025 Improper Output Handling Whitepaper
  12. LLM01:2025 Prompt Injection Whitepaper
  13. LLM10:2025 Unbounded Consumption Whitepaper
  14. AI Risk Management Framework (NIST AI 100-1) Whitepaper
  15. NIST AI 600-1: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile Whitepaper
  16. OWASP Top 10 for LLM Applications 2025 Whitepaper
  17. ISO/IEC 42001:2023 AI management systems Whitepaper
  18. CompTIA SecAI+ (CY0-001) certification