AI App Security Checklist: Evidence to Collect Before Release

What an AI app security checklist must cover

An AI app combines conventional software risks with model-specific uncertainty. Authentication can fail in ordinary ways, while untrusted content can also influence a model, retrieved documents can cross tenant boundaries, and an agent can attempt actions its user never intended. A useful review must examine the whole application—not treat a successful chatbot conversation as proof of security.

This checklist draws on established sources including the NIST AI Risk Management Framework, NIST Secure Software Development Framework, OWASP Application Security Verification Standard, OWASP Top 10 for Large Language Model Applications, OWASP API Security Top 10 and MITRE ATLAS. These are reference points, not certificates. Select controls according to the app’s architecture, users, data and potential impact, and check the current versions of each source.

Define the system and trust boundaries.
Test conventional application and API security.
Test model, retrieval and agent-specific failure modes.
Preserve reproducible evidence rather than relying on verbal assurance.
Record residual risk, ownership and the release decision.

1. Inventory the real system before testing it

Start with the deployed application and trace every component that can receive data or cause an effect. Include the browser or client, backend APIs, identity provider, model providers, vector stores, retrieval pipelines, file processors, analytics, plugins, tools, queues and human approval steps. Note which components are production, staging or shared.

Model names alone are insufficient provenance. Record the provider, model or deployment identifier, configuration, system instructions, safety settings, tool definitions and evaluation version where available. Hosted models may change outside your release cycle, so distinguish controls you operate from assumptions about a provider.

Create a data-flow diagram with trust boundaries and external services.
List model, prompt, retrieval, tool and policy versions used in the tested build.
Identify secrets, personal data, confidential content and regulated or contract-sensitive data.
Document retention, deletion and provider-training settings as configured—not as presumed.
Mark unresolved facts as unknown rather than converting them into pass results.

2. Build a threat model around assets and actions

Threat modeling should begin with what can be exposed, altered or executed. Consider outsiders, authenticated users, malicious tenants, compromised dependencies, poisoned data sources and accidental misuse. For agentic applications, separate the authority to suggest an action from the authority to execute it.

Map credible abuse paths: extracting another customer’s documents, overriding instructions through retrieved content, manipulating an agent into sending a message, exhausting model spend, exposing secrets in logs or causing an unsafe decision through fabricated output. Rate severity using your organization’s method, but retain the assumptions behind each rating.

Name the asset, attacker or failure source, entry point and potential consequence.
Include indirect prompt injection from webpages, files, emails and retrieved records.
Model confused-deputy risks where the app has more authority than the user.
Treat denial-of-wallet and resource exhaustion as availability and cost risks.
Assign an owner and planned control to every material threat.

3. Verify identity, authorization and tenant isolation

A model must not decide whether a user is authorized. Enforce authentication and authorization in deterministic application code at every protected API, retrieval request and tool invocation. Verify object-level access rather than checking only whether the requester is signed in.

Test with at least two isolated users or tenants and controlled records. Attempt direct API calls, modified object identifiers, stale sessions, role changes and access after revocation. Use synthetic or disposable test data where practical, and obtain explicit permission before testing any system you do not own.

Require server-side authorization for every sensitive object and action.
Scope retrieval indexes, caches, conversation memory and exports by tenant and user.
Use short-lived credentials and narrowly scoped service identities.
Protect administrative and support functions with stronger controls.
Retain request, decision and response evidence with secrets redacted.

4. Test prompts, retrieval and untrusted content

Prompt injection cannot be addressed by a single hidden instruction. Treat prompts, uploaded files, retrieved passages, tool output and webpages as untrusted inputs. The model may follow hostile content even when it is labelled as data, so security boundaries must exist outside the model.

Run adversarial tests that reflect actual product capabilities. Try instruction conflicts, encoded or obfuscated requests, malicious documents, poisoned retrieval results and attempts to reveal hidden context. A refusal is useful evidence only for the tested configuration and input; it does not establish universal resistance.

Separate trusted instructions from untrusted content in architecture and processing.
Restrict which data can enter context and minimize unnecessary sensitive context.
Apply access control before retrieval, not after generation.
Validate and constrain model output before rendering or passing it to another system.
Test whether citations refer to the retrieved source and support the specific claim.
Record prompts, retrieved passages, configuration, outputs and timestamps for replay.

5. Constrain tools and consequential actions

Tool-using agents create the clearest path from unreliable output to real-world impact. Give each tool the least privilege needed, validate structured arguments against an allowlist or schema, and recheck authorization at execution time. Never assume the model’s stated intent is trustworthy.

Separate planning, approval and execution. Actions such as publishing, purchasing, deleting, changing permissions, sending external communications or modifying production data should use explicit confirmation or an appropriate human approval gate. Design idempotency and reconciliation according to the underlying service; do not assume a model-orchestrated retry produces exactly-once effects.

Use read-only tools by default and isolate write capabilities.
Bind tool access to the current user, tenant, task and session.
Display the exact target, parameters and consequence before approval.
Stop safely on unknown targets, ambiguous instructions or missing authority.
Test cancellation, timeout, partial failure, duplicate requests and rollback paths.

6. Protect data, secrets and model interfaces

Minimize what the application sends to models and what it stores afterward. Redact or tokenize sensitive fields where that still supports the use case. Confirm how prompts, outputs, embeddings, feedback and traces are retained across your own systems and external providers.

Do not place secrets in system prompts, client code or model-visible context. Store credentials in an appropriate secret manager, rotate them, and prevent sensitive values from entering logs. Apply transport encryption and suitable storage protection, while recognizing that encryption does not correct excessive access.

Document data purpose, location, recipients, retention and deletion behavior.
Test file uploads for type, size, malware and parser-related abuse.
Prevent credentials and internal tokens from appearing in model context or output.
Review logs, traces, analytics and support tools for accidental disclosure.
Verify deletion and account closure behavior with controlled records.

7. Secure code, APIs and the AI supply chain

AI features do not replace baseline application security. Review injection, cross-site scripting, cross-site request forgery, server-side request forgery, insecure deserialization, broken access control, misconfiguration and vulnerable dependencies according to the technologies in use. Apply rate limits and request limits at APIs rather than relying on model refusals.

Track provenance for packages, model artifacts, datasets, prompts, evaluation suites and hosted services. Pin versions where feasible, verify artifact integrity, scan dependencies and container images, and define how provider or model changes trigger reevaluation. Generated code deserves the same review and testing as human-written code.

Maintain software and model component inventories.
Lock and review dependencies; remove unused packages and permissions.
Protect outbound network access against arbitrary or internal destinations.
Scan source and deployment configuration for exposed secrets.
Require code review, automated tests and controlled deployment for AI-generated changes.