What counts as AI security?
AI security covers more than protecting a model endpoint. It includes the data supplied to a model, retrieved knowledge, system instructions, tool permissions, generated outputs, application code, third-party dependencies and the people who approve consequential actions. A secure model inside an insecure workflow can still expose data or perform an unauthorized operation.
The examples below reflect recurring risk categories described in resources such as the NIST AI Risk Management Framework, OWASP guidance for large language model applications and MITRE ATLAS. These frameworks help organize threats, but they do not prove that a particular application is safe. That requires testing the real implementation, recording what happened and interpreting results in context.
Two meanings are often mixed together: securing systems that use AI, and using AI to support security work. This article concentrates on the first because it creates application-specific failure modes. AI-assisted alert triage or code review can help defenders, but those outputs still require validation.
1. Indirect prompt injection through untrusted content
Consider an assistant that reads support tickets, web pages or uploaded documents. An attacker hides instructions in that content: ignore prior rules, reveal internal context or send information to an external destination. The user never types the malicious prompt; retrieval introduces it. The model may treat data as instructions unless the surrounding application enforces a trustworthy boundary.
A useful test places benign but adversarial instructions inside every content channel the application accepts. Observe whether the system changes goals, reveals protected context, invokes tools or asks for approval. Test different formats because controls that work for plain text may fail with HTML, document metadata or image-derived text.
Mitigations include isolating instructions from retrieved data, constraining tools outside the model, filtering inaccessible content and requiring approval for consequential operations. A prompt telling the model to be careful is not an authorization control.
2. Sensitive information disclosure
An AI feature may expose secrets through its response, logs, retrieval results or debugging traces. For example, a customer-support assistant could retrieve another tenant’s record because the vector search applies a broad filter. A coding assistant might receive environment secrets that were unnecessarily included in its context.
Test with accounts from at least two roles or tenants and create distinctive disposable records for each. Ask direct questions, use paraphrases, manipulate retrieval context and inspect network traffic and logs. The evidence should establish which identity made the request, which records were retrieved and what reached the model—not merely what appeared in the final answer.
Controls belong at the data layer: tenant-aware retrieval, least-privilege service identities, secret redaction, minimal context construction and retention rules. Output filtering can provide defense in depth, but it should not compensate for unauthorized retrieval.
3. Excessive agency and unsafe tool use
An agent that can send email, change records, issue refunds or deploy code creates a different risk from a chatbot. A plausible failure is a model interpreting an ambiguous request as authority to take an irreversible action. Another is a compromised data source steering the agent toward a tool call the user did not intend.
Test the boundary between planning and execution. Ask for hypothetical actions, introduce ambiguity, exceed monetary or scope limits, withdraw approval and make required information unavailable. The system should stop safely when intent, authority or target state is uncertain. Confirm enforcement in application code or policy infrastructure rather than relying only on model wording.
Strong patterns include narrowly scoped tools, typed parameters, server-side authorization, previews, explicit confirmation, idempotency where feasible and human approval for defined high-impact actions. Exactly-once side effects should not be assumed: retries, timeouts and partial failures need deliberate handling.
4. Poisoned retrieval data or knowledge sources
Retrieval-augmented generation can make answers more relevant, but it also creates an ingestion boundary. An attacker—or an accidental publishing process—could introduce a document that falsely claims to be authoritative, embeds malicious instructions or outranks a trusted source.
Create test documents with known provenance, conflicting claims and different access permissions. Check whether ingestion preserves source identity, freshness and ownership; whether retrieval respects permissions; and whether the answer identifies uncertainty when sources disagree. Deleting or revoking a document should also remove its practical influence within an understood timeframe.
Useful controls include allowlisted sources, signed or attributable artifacts, ingestion review, access-aware indexing, citation links, version tracking and re-indexing procedures. Citations improve inspectability, but a citation is not proof that the source is correct or authorized.
5. Model, plugin and software supply-chain compromise
AI applications depend on model providers, open-source packages, model files, browser components, plugins and data pipelines. A compromised dependency could execute code, alter behavior or transmit information. A provider model update can also change application behavior without any source-code change in the app.
Inventory model names and versions where available, providers, packages, tools, data sources and deployment artifacts. Pin versions when operationally possible, verify checksums or signatures, scan dependencies and retest important journeys after changes. Where a hosted provider does not expose an exact model revision, record that limitation rather than inventing precision.
Sandboxing, network restrictions, minimal credentials and controlled update processes reduce exposure. A software bill of materials can help, but AI systems also need an account of models, prompts, retrieval sources and external tools.
6. Resource exhaustion and denial of wallet
AI usage often carries variable computational cost. Attackers may submit unusually long inputs, trigger recursive tool use, request large files repeatedly or create many concurrent sessions. The service may remain technically available while accumulating unacceptable latency or expense.
Test size, rate, concurrency and recursion limits using authorized, bounded workloads. Verify that one tenant cannot consume another tenant’s capacity and that cancellation actually stops downstream work. Record request counts, token or compute consumption, tool calls, elapsed time and the point at which controls intervene.
Defenses include authenticated quotas, payload limits, bounded agent steps, timeouts, caching, circuit breakers and cost alerts. Limits should be enforced before expensive processing where possible.
7. Insecure handling of generated output
Model output is untrusted input to the rest of the application. If generated HTML, SQL, shell commands or file paths are executed without validation, ordinary model errors or manipulated prompts can become conventional vulnerabilities such as cross-site scripting, injection or path traversal.
Test generated content with hostile markup, command-like strings, malformed structured data and unexpected encodings. Follow the output to its final sink: a browser, database, terminal, API or document renderer. A response that looks harmless in a chat panel may become dangerous when another component interprets it.
Use contextual encoding, parameterized queries, schema validation, sandboxed rendering and explicit allowlists. The model should propose data or actions; trusted code should decide how they are represented and whether they may execute.