EN हि ES FR AR
SOC 2 · ISO 27001 · Data residency on request
A.26Security

Prompt Injection and AI Agent Security: How to Protect Enterprise AI Systems

Practical safeguards for AI systems that read untrusted content, retrieve company data, and use tools.

An AI agent can read an email, search a knowledge base, call an API, and prepare a business action. That combination is useful, but it creates a security boundary that ordinary chat interfaces do not have. Prompt injection is an attempt to steer the model through instructions placed in a prompt or in content it reads. A secure design assumes that some instructions will be misleading, then limits what the system can access or do if the model follows them. Prompt injection cannot be reliably removed with a clever system prompt alone.

What prompt injection is

Language models process instructions and ordinary text in a shared context. A direct prompt injection arrives in a user’s message, such as a request to ignore the task and disclose information. An indirect injection is placed in content the system later reads, such as a web page, email, document, support ticket, or code comment. The text may look like normal content to a person while trying to influence the model. The application cannot safely assume that retrieved material is trustworthy just because it came from an approved search index. OWASP prompt injection risk

This differs from a classic database injection flaw. SQL can separate executable commands from data through structured query mechanisms. Natural-language models are designed to interpret language, and attackers can exploit that same capability by blending instructions with data. Filters and system prompts can reduce some risks, but they do not create a guaranteed boundary between trusted instructions and untrusted text. OWASP’s LLM01 entry notes that retrieval and fine-tuning do not fully mitigate prompt injection, and that it is unclear whether foolproof prevention methods exist. OWASP prompt injection risk

The consequence depends on the surrounding application. A text-only assistant might return a misleading answer. A connected assistant might expose content it can retrieve. An agent with tools might also send a message, alter a record, or call an API. The model does not need to be “hacked” in the traditional sense for a workflow to fail. The security question is whether untrusted input can influence a sensitive decision, and whether the software independently checks the data access and action that follow.

What documented cases teach us

In 2025, CVE-2025-32711 was published for Microsoft 365 Copilot. The National Vulnerability Database describes an AI command-injection issue that could allow an unauthorized attacker to disclose information over a network. NVD lists a Microsoft security advisory among its references. This is evidence of a documented vulnerability disclosure, not by itself evidence that attackers exploited it in the wild. The lesson for organizations is to treat assistant features that process messages and connected information as software with a threat model, permissions, updates, and incident ownership. NVD record for CVE-2025-32711

OWASP’s prompt-injection prevention guidance also points readers to research on remote prompt injection in GitLab Duo. That case illustrates why content supplied by another user, a repository, or an external source must be treated as untrusted when an assistant summarizes or acts on it. A useful review asks what the assistant is allowed to see, whether external text can change its task, and whether a suggestion can reach a tool without a separate authorization check. Security research demonstrations help teams build tests, but a demonstrated attack path should not be described as a confirmed production incident unless a reliable source establishes that fact. OWASP prompt-injection case references

These examples do not mean every AI assistant is exposed in the same way. Product configuration, available connectors, user identity, and the application’s action controls all matter. They do show why prompt injection belongs in threat modeling and security testing. The same document can be harmless when summarized in a read-only tool and dangerous when an agent can use it to initiate privileged actions.

Map an attack as a sequence rather than looking for one suspicious phrase. First, identify where untrusted content enters. Then follow what the model can retrieve, which tools the application exposes, how the proposed action is checked, and where the result is sent. A control at each boundary can limit the next step. For example, an injected instruction may influence a draft, but it should not be able to select an arbitrary recipient, read a restricted document, or send the draft without the application’s authorization checks. This chain-based view also gives incident responders useful evidence to collect.

Common risks for assistants and agents

Unauthorized access and sensitive data exposure

If an assistant retrieves content using a broad service account, it may expose information beyond the current user’s authority. A prompt injection can try to make the assistant search for secrets or place private material in its response. The underlying problem is excessive access, not only unreliable model behavior. Enforce authorization at retrieval time using the requesting user’s identity or an equivalent permission-aware check. Do not rely on the model to decide whether a person is allowed to see a document.

Actions taken without valid authority

An agent may have tools to create tickets, update customer records, send email, or change account settings. A malicious instruction can try to redirect those tools to an unintended target or operation. A natural-language refusal is not an access-control mechanism. The API must check the caller, resource, action, and business rules independently. Keep read, draft, and write permissions separate so that a summarization task cannot quietly become an account-management task.

Data leakage through outputs and external destinations

An assistant could include confidential text in a generated answer, a tool argument, a message, or a request to an external service. Review all output paths, including logs, traces, notifications, and connected systems. Apply data-loss rules where appropriate, but do not assume a filter will recognize every sensitive value or every encoding. Limit what data the model receives in the first place, and validate destinations and recipients in application code before sending information outside the organization.

Excessive agency and unclear accountability

“Agency” is the ability of a system to take actions through connected tools. An agent with broad, persistent permissions can turn a small model error into a larger operational problem. If several tools are chained together, it may be difficult to identify which decision led to an action unless the system records each step. Define an owner, a permitted task, and a stopping condition. Keep actions reversible where possible and make the user or reviewer responsible for high-impact decisions.

Best practices to secure AI agents

Apply least privilege outside the model

Give each agent a dedicated identity with only the permissions needed for its approved task. Prefer short-lived credentials and narrow scopes. Enforce authorization in the API or integration layer, not in the prompt. Validate every tool call against an allowlist of operations and permitted resources. A model can propose a call, but ordinary software must decide whether the call is allowed. Separate development, test, and production credentials, and make revocation straightforward.

As an illustrative example, a support agent that drafts ticket replies may need read access to a single ticket and approved help content, but not permission to export a customer database or change a user’s access. If the task later expands, review the new permission as a design change. Do not add a powerful account because it is simpler to configure. The smallest permission set makes both accidental and adversarial actions easier to contain.

Imagine an onboarding agent that reads a new employee’s checklist and prepares account requests. A safe design can let it read approved onboarding material and create a draft request, while a manager confirms the employee, role, and access before a separate identity system applies the change. A hostile instruction inside an uploaded document should not grant new access or change the approval path. The example is illustrative, but the control pattern is reusable: constrain what the agent can propose, verify the request in the system that owns the record, and keep the final authority with an accountable role.

Keep untrusted content separate and retrieval permission-aware

Label external and user-supplied content as untrusted in the application’s data model. Keep system instructions, authorization decisions, and secrets out of retrieved documents and model-visible context wherever possible. Use retrieval filters tied to the user’s permissions, and check those permissions again before returning content. In an illustrative case, a project assistant searches internal files for a manager. It should retrieve only documents that manager can open directly, even if an indexed file contains instructions telling the assistant to ignore its rules. APLS guide to permission-aware retrieval

Require human approval for consequential actions

Place an approval step between the agent and actions that send external communications, move money, disclose sensitive data, grant access, or make difficult-to-reverse changes. Present the reviewer with the proposed action, the destination, the data involved, and the reason it is being requested. Approval should be a real authorization check in the application, not a phrase the model can generate. For low-risk tasks, use limits, rate controls, and reversible drafts rather than asking a person to approve every minor step.

Validate tool inputs and outputs deterministically

Define strict schemas for tool arguments and reject unknown fields, invalid identifiers, and operations outside the task. Use server-side checks for account ownership, allowed recipients, transaction limits, and state transitions. Treat generated code, links, queries, and commands as untrusted until reviewed and tested. If an output is used by another system, encode it for that destination and apply the receiving system’s normal validation. A model’s confidence or explanation is not proof that its output is safe.

Design a safe failure path. If an approval service is unavailable, a permission check fails, or the agent receives contradictory instructions, it should stop or fall back to a read-only response rather than guess. Make retries idempotent so an uncertain result does not trigger repeated writes. Keep a person able to inspect a proposed action before it is sent, and record the reason a request was rejected or paused. These ordinary reliability controls matter because a security boundary that disappears during an outage is not a dependable boundary.

Test adversarially and monitor behavior

Build tests from realistic sources of untrusted text: an email attachment, a public web page, a customer comment, an image, or a document with conflicting instructions. Check whether the system reveals restricted content, calls tools outside its scope, or skips required approval. Test different user roles and access changes, not just a single administrator account. OWASP’s prevention guidance recommends regular security testing with known attack patterns and a defense-in-depth design, and describes input screening and structured prompts as layers, not a complete defense. OWASP testing and mitigation measures

Log the user, retrieved sources, model and prompt versions, tool name, validated arguments, approval, and final result where policy and privacy requirements allow. Avoid indiscriminately storing sensitive prompts. Alert on unusual retrieval, repeated denied actions, unexpected destinations, permission changes, or a sudden increase in tool use. Give operators a way to pause the agent, revoke its credentials, and switch to a safe manual process. NCSC guidance treats secure operation and maintenance as part of the AI system lifecycle. NCSC secure AI system development guidance

A practical AI agent security checklist

  • Document the agent’s owner, purpose, users, data sources, tools, and prohibited actions.
  • Classify every source as trusted, user-supplied, or external, and treat retrieved instructions as untrusted.
  • Enforce user-level authorization on every retrieval and API operation.
  • Use dedicated identities, narrow scopes, short-lived credentials, and easy revocation.
  • Require clear human approval for sensitive, external, or hard-to-reverse actions.
  • Validate tool arguments, destinations, outputs, and business rules in deterministic code.
  • Test direct and indirect prompt injection across roles, documents, links, and connected tools.
  • Monitor retrieval and tool actions, preserve useful audit evidence, and rehearse a pause-and-revoke response.

Prompt injection is a risk to manage, not a problem that can be declared solved. Put security boundaries in identity systems, retrieval controls, API checks, and human approval. Let the model help interpret information, but keep authority over data and actions in software and accountable people.

For the broader operating model, read APLS guidance on bounded enterprise AI agents and review agent governance capabilities.

← All insights Start a conversation →