Start by defining what must be tested
To answer how to test an AI-generated app before launch, first distinguish between an app whose code was generated with AI and an app that also uses an AI model at runtime. Both require conventional software testing. Runtime AI features need additional evaluation because outputs may vary with prompts, model versions, retrieved data and tool access.
Do not try to prove that the app is bug-free. Define the release candidate, identify credible failures and collect enough evidence to make a bounded launch decision. Freeze or label the exact build under review so results are not silently invalidated by later changes.
- Record the commit, deployment identifier, environment and relevant configuration.
- List user roles, sensitive data, external services and consequential actions.
- Separate launch blockers from issues that can be accepted temporarily by a named owner.
- Retest after material code, prompt, model, permission or infrastructure changes.
Create an evidence and provenance record
A pass or fail without provenance is difficult to review. For every test, retain the preconditions, input, expected result, actual result, timestamp and release version. Add screenshots, logs, traces or source references where they help, while excluding credentials and unnecessary personal data.
AI evaluations need more context. Record the model and version when available, system instructions, tool definitions, retrieval source, sampling parameters and seed if the provider supports one. A seed may improve reproducibility, but it does not guarantee identical results across model or infrastructure changes. Likewise, a score such as 339/1000 is not meaningful on its own: retain the rubric, evaluator, test set, sample size, threshold and individual failures.
- Preserve raw evidence rather than only a rewritten summary.
- Link each finding to the affected requirement and release version.
- Document unavailable evidence and unresolved uncertainty explicitly.
Test complete journeys on the deployed candidate
Unit tests and generated previews cannot show that a deployed application works end to end. Exercise the real HTTPS release candidate through the browser, using controlled accounts and disposable data. Verify the resulting state in the interface, backend and connected services where you have authorized access.
Cover the main success path, but spend equal attention on interruption and recovery. Refresh midway, submit twice, navigate backward, allow a session to expire and simulate a slow or unavailable dependency. Check whether partial work is duplicated, lost or left in an unsafe state.
- Onboarding, sign-in, sign-out and account recovery.
- Create, read, update, archive and delete flows as applicable.
- Validation for malformed, empty, oversized and duplicate inputs.
- Cancellation, retry, timeout and partial-failure behavior.
- Emails, exports, webhooks or payments only in approved test or sandbox conditions.
Challenge identity, permissions and consequential actions
Test every supported role independently. A normal user should not gain administrative access by changing a URL, object identifier, request body or client-side state. Confirm that authorization is enforced by the server and that records remain isolated between accounts or organizations.
Treat deletion, publication, billing, outbound communication, permission changes and external tool calls as consequential. Use sandbox integrations, disposable records and explicit human confirmation. If the effect cannot be safely contained or reversed, stop rather than experimenting against a live target. Also test repeated requests and retries; design for idempotency where appropriate, without assuming exactly-once side effects.
- Attempt direct access to another role’s routes and resources.
- Check expired, revoked and missing credentials.
- Verify confirmation, audit and cancellation behavior for high-impact actions.
- Confirm that unknown tool requests fail closed instead of being guessed or executed.
Evaluate AI behavior as a system, not a demo
A few convincing prompts are not a reliable evaluation. Build a versioned set of representative, difficult and prohibited cases based on the app’s intended use. Run them repeatedly when outputs are variable, then inspect the distribution of failures rather than reporting only an average score.
Define what can be judged automatically and what requires a qualified human. Exact-format checks may be deterministic; factual support, harmful advice or domain quality may not be. Where the model uses retrieval or tools, test the whole chain: source permissions, citation or attribution behavior, prompt-injection resistance, tool arguments and the handling of unavailable information.
- Normal requests, ambiguous requests and unsupported assumptions.
- Attempts to override system instructions or expose hidden data.
- Hostile content embedded in uploaded or retrieved material.
- Malformed model output, excessive latency and provider failure.
- Output escaping before model content reaches HTML, code, queries or commands.
- Clear uncertainty and human escalation for cases outside the product’s limits.
Inspect source, configuration and data handling
AI-generated code can be plausible while containing unused dependencies, duplicated logic, unsafe defaults or missing authorization checks. Review the source and generated configuration when available. Confirm that the deployed artifact corresponds to the reviewed revision; a clean repository does not establish what is actually running.
Scan for exposed secrets and vulnerable dependencies, but do not treat a scanner as a complete security review. Inspect authentication, session handling, input validation, file uploads, server-side requests and database access in context. Map what data is collected, where it is sent, how long it is retained and whether deletion behaves as described. Obtain appropriate specialist or legal review for obligations that automated tests cannot establish.
- Keep secrets out of repositories, browser bundles, prompts and logs.
- Pin and review dependencies, build scripts and deployment settings.
- Test tenant separation and least-privilege access to storage and integrations.
- Verify that logs and analytics do not capture sensitive content unnecessarily.
Cover accessibility, usability and operational failure
A technically correct journey can still be unusable. Test with a keyboard, visible focus, zoom, reduced viewport sizes and at least a screen-reader smoke test. Review labels, instructions, errors and status changes. Automated accessibility checks are useful for coverage, but they cannot by themselves establish accessibility or conformance.
Set explicit performance and reliability budgets appropriate to the app, then measure them on representative devices and networks. AI calls can be slow or unavailable, so the interface should expose progress, permit cancellation where feasible and recover without duplicate effects.
- Check loading, empty, error, offline and permission-denied states.
- Test supported browsers, mobile layouts and long generated content.
- Verify rate-limit, quota and dependency-outage behavior.
- Ensure critical information is not conveyed by color alone.
Make launch readiness a reviewable decision
Before launch, assemble the evidence into a concise decision record. Every required subject area should have a result: passed with evidence, failed, accepted with a named owner and deadline, or not tested with an explanation. A single aggregate score can hide a severe authorization or data-loss problem, so retain the underlying findings.
Operational readiness belongs in the same review. Confirm monitoring for user-visible failures, model or dependency errors and unusual action rates. Rehearse rollback or feature disablement, document who can act during an incident and verify that support can recognize known failure modes.
- The reviewed build matches the intended release.
- Critical journeys pass with controlled test accounts and data.
- Role boundaries and consequential actions have been challenged.
- AI evaluations are versioned and retain provenance.
- Source, dependencies, secrets and data flows have been inspected.
- Accessibility and failure states have received human review.
- Monitoring, rollback, escalation and reinspection are ready.
- Unresolved risks have explicit owners rather than optimistic assumptions.