Release readiness is an evidence claim, not a feeling
An AI app is not ready merely because its main demo works. Release readiness means the identified release candidate has passed defined checks, material findings have owners and dispositions, and decision-makers understand what remains uncertain.
AI adds nondeterministic outputs, model and prompt dependencies, data-handling questions, misuse paths and potentially consequential tool actions. It does not replace conventional software concerns such as authorization, accessibility, dependency risk, error recovery and rollback. A useful AI app release readiness checklist must cover both.
Treat every pass as a claim requiring provenance: what was tested, against which version, in which environment, with what data, by whom or what, when, and with what raw evidence. A screenshot without that context is weak evidence.
1. Freeze the release candidate and scope
Start by identifying the exact candidate under review. Record the application URL and environment, source commit or build identifier, deployment time, model and provider, prompt or agent configuration, enabled tools, feature flags, dependency lockfile and relevant data or retrieval snapshot.
Define what the decision covers. List supported users, critical journeys, integrations, browsers and devices, excluded features, and known environmental differences. If code, prompts, models, permissions or infrastructure change materially after review, mark the affected evidence stale and reinspect rather than silently carrying the result forward.
2. Use this concise AI app release readiness checklist
For each item, record a result, linked evidence, owner and residual uncertainty. “Not applicable” should include a reason rather than act as an escape hatch.
- Release identity is fixed and traceable to source, configuration, models and deployment.
- Critical user journeys have explicit acceptance criteria and realistic test data.
- AI quality is evaluated on representative, provenance-recorded cases, including difficult inputs.
- Repeated trials expose output variability where nondeterminism could affect the result.
- Prompt injection, unsafe content, data leakage and tool misuse scenarios are exercised.
- Authentication and server-side authorization are checked for users, roles, objects and actions.
- Consequential actions require appropriate confirmation, permissions and safe failure behavior.
- Privacy, retention, logging and third-party data flows are documented and reviewed by accountable owners tied to the applicable context and obligations, not assumed compliant from a checklist pass alone. Accessibility includes keyboard use, focus, labels, status announcements and error recovery; WCAG can provide testable criteria where relevant. Reliability evidence covers timeouts, retries, partial failures, rate limits and dependency outages. Observability can detect user-impacting failures without unnecessarily exposing sensitive data. Rollback, feature-disable and incident-response procedures are documented and rehearsed where practical. Every material finding has severity, evidence, an owner, a disposition and a retest state. A named decision-maker records go, conditional go or no-go with accepted residual risk.
3. Evaluate the AI behavior, not just the happy-path answer
Build an evaluation set from the app’s intended tasks, foreseeable edge cases and credible misuse. Record the origin and permitted use of each case. Use exact expected outputs only where appropriate; otherwise define a rubric with observable pass conditions. Separate factuality, relevance, safety, format adherence, citation quality and refusal behavior rather than hiding them inside one average score.
Run enough repeated trials to reveal meaningful variation, especially for high-impact steps. Retain prompts, retrieved context, model and configuration identifiers, outputs, tool calls, evaluator decisions and timestamps. If a hosted model can change outside your deployment, state that limitation and define when reevaluation is triggered.
Exercise indirect and direct prompt injection, hostile uploads, malformed input, unsupported requests, sensitive-data extraction attempts and attempts to exceed tool authority. Frameworks such as the NIST AI Risk Management Framework and OWASP guidance for generative AI can help identify questions, but referencing a framework is not evidence that the app passed them.
4. Test the complete product around the model
Review the end-to-end journey from account creation through completion, cancellation and recovery. Check empty, loading, error, timeout and degraded states. Confirm that model failures are translated into useful user choices instead of fabricated success, endless spinners or lost work.
Test authorization on the server side, including cross-account object access and privileged actions. Inspect secret handling, dependency provenance, security headers and logging. For side effects such as messages, purchases, deletions or external updates, verify confirmations, replay behavior and recovery. Do not infer exactly-once execution from a single successful run.
Accessibility evidence should include keyboard operation, visible focus, meaningful names, reading order, contrast, validation messages and announcements for dynamically generated results. Test narrow screens and zoom. Automated scans can identify some issues, but they do not replace manual interaction and assistive-technology checks.
5. Execute journeys with explicit authority and safe boundaries
Before a reviewer or agent interacts with a live application, establish separate gates: review capacity, entitlement to test the target, disposable accounts and data, authority for each action, and human availability when supervision or approval is required. Prefer isolated environments, but do not assume a staging environment faithfully represents production.
Bind approval to a specific journey and version. Map each step to its possible side effects, required permissions and stop conditions. Unknown or consequential actions should stop safely rather than being guessed through. Examples include accepting legal terms, spending money, publishing content, deleting records, contacting real people or changing production access.
Keep an execution record that distinguishes observation from inference. A visible success message may show what the interface reported; it does not by itself prove that a provider accepted a resource, that a downstream system completed work, or that retries cannot duplicate an effect.
6. Preserve raw evidence and make findings reviewable
Store raw evidence immutably where feasible: request and response details, console output, screenshots, recordings, source references, test inputs and tool traces, with sensitive material minimized or redacted under a documented policy. Derived summaries may be edited for clarity, but edits should not overwrite the underlying record.
Normalize and deduplicate findings without erasing independent corroboration. Each finding should contain the affected release, requirement or risk, reproduction steps, observed evidence, expected behavior, severity rationale, uncertainty, owner and status.
- Open: unresolved and relevant to the decision.
- Accepted risk: explicitly owned, justified and time-bounded where appropriate.
- Remediated: changed but not yet confirmed against the new candidate.
- Verified: reinspected with fresh evidence.
- Invalid or duplicate: closed with a reason and links to the preserved evidence.
7. Make the go/no-go decision explicit
Avoid turning readiness into an unexplained percentage. A score can conceal a release-blocking authorization flaw beneath many low-value passes. Define blocking conditions before testing and evaluate findings by user impact, exploitability or likelihood, detectability, reversibility and scope.
A go decision means the agreed criteria are met and residual risks are explicitly accepted. A conditional go names the conditions, owner, deadline, monitoring and fallback. A no-go identifies the blockers and evidence required for reconsideration. None of these labels guarantees security, compliance, accessibility, commercial results or a successful release.
Record who decided, the evidence reviewed, unresolved disagreements, assumptions, expiry conditions and rollback trigger. Reopen the decision when material code, model, prompt, data, permissions or infrastructure changes invalidate prior evidence.