QA report: external/prompt.always200.com at hosted
Model configuration saves without required API key validation, and empty evaluation runs are permitted.
The test run evaluated 12 scenarios covering core hash routing, model and variable configuration workflows, state persistence, and empty-state validations. Ten scenarios passed, one failed, and one was blocked when scenario S2 reached its tool-call budget before the final submission and verification step could be executed.
Two functional issues were identified involving input validation. In the Model Manager, entering a display name enables the save action and allows configurations with empty API keys to be persisted without validation (F1). In addition, the application inconsistently enforces empty-state validation across primary controls: while prompt optimization is disabled when empty, the Run All button permits submission with empty test content, producing evaluations that result in no content (F2).
Core routing and variable management workflows remain stable, but the missing form and submission validations permit incomplete provider setups and empty execution runs.
Run summary
| Metric | Count |
|---|
| Scenarios executed | 12 |
| Passed | 10 |
| Failed | 1 |
| Blocked | 1 |
| Findings raised | 1 |
| Issues after the audit | 2 |
| Withdrawn by the audit | 0 |
| Critical / high / medium / low | 0 / 0 / 1 / 1 |
Target: https://prompt.always200.com · Testing level: deep_feature · Stack: unknown
Issues
Medium severity
F1 · Model Manager saves provider configuration with missing API Key without validation
Severity: medium · Type: functional · Verdict: confirmed · Scenario: S3
The evidence supports the claim that the Create button becomes enabled and allows saving the model configuration even when the API Key field is left empty.
Expected: The application should require an API Key for remote providers (such as OpenAI) and display a validation error when attempting to save with an empty API Key.
Actual: The Create button is enabled once Display Name is populated, allowing the user to save a model with an empty API Key without validation.
Steps to reproduce:
- Navigate to https://prompt.always200.com/#/basic/system
- Click the Model Manager button
- Click Add
- Enter a Display Name and leave the API Key field empty
- Click Create
Evidence: screenshots/S3-3.png, screenshots/S3-6.png
Low severity
F2 · Run All button allows submission of empty test content
Severity: low · Type: functional · Verdict: promoted · Scenario: S10
Surfaced by the audit of S10, which the Tester passed, and reproduced live: The 'Run All' button is enabled even when the 'Test Content' field is empty. Clicking it does not throw an error or prevent execution, but results in 'No content' being displayed for the evaluation output.
Expected: The 'Run All' button should be disabled when the Test Content is empty, consistent with the validation on the Optimize button.
Actual: The 'Run All' button is enabled and accepts the empty submission, executing an evaluation that results in 'No content'.
Steps to reproduce:
- Navigate to /#/basic/system
- Ensure Test Content field is empty
- Click the 'Run All' button
Evidence: screenshots/critic-M1-1.png
Scenario results
| Scenario | Priority | Result | Issues |
|---|
| S1 Verify hash routing across core views | high | pass | none |
| S2 Model Manager - Create valid model configuration | high | blocked (Tool-call budget was reached before the final submission and verification step could be executed.) | none |
| S3 Model Manager - Required fields validation | high | fail | F1 |
| S4 Variable Manager - Create custom variable | high | pass | none |
| S5 Variable Manager - Required field validation | high | pass | none |
| S6 Model Manager - Edit existing model | medium | pass | none |
| S7 State Persistence - Model configuration survives page reload | medium | pass | none |
| S8 Variable Manager - Modal cancellation | medium | pass | none |
| S9 Optimize action empty state validation | medium | pass | none |
| S10 Run All execution empty state validation | medium | pass | F2 |
| S11 Favorites page empty state | low | pass | none |
| S12 Model Manager - Optional advanced fields | low | pass | none |
The audit
The Critic reviewed 1 finding and re-verified 1 of them live in the browser, replaying the reported steps on a fresh page.
- Scenario S2 was blocked because the tool-call budget was reached before the test could verify the created model.
- The application inconsistently applies empty-state validation across its primary actions (disabling 'Optimize' for empty prompts but allowing 'Run All' for empty test content).
What to fix first
- Enforce API key validation in the Model Manager before enabling the save action (F1).
- Disable the Run All action when test content is empty to ensure consistent validation across primary controls (F2).
Coverage and caveats
In scope: Hash routing across core views; Model Manager configuration form and validation; Variable Manager configuration form and validation; Local state persistence for configured models; Modal dialog behavior and cancellation; Empty state validation for main action buttons.
Not covered: End-to-end prompt optimization execution with real LLM responses (requires paid API keys, out of scope for a hosted run); Image generation test executions (requires real API keys, out of scope for a hosted run).
- The tester does not possess valid API keys for LLM providers, so mock keys are used to test configuration persistence without executing real network calls to the providers.
- Browser local storage is not cleared between scenarios in the same session, enabling persistence tests.
- The application relies entirely on client-side routing, so page navigation will use the hash URLs provided in the AppMap.
- S2 could not be executed: Tool-call budget was reached before the final submission and verification step could be executed..
By the numbers
| Metric | Value |
|---|
| Scenarios | 10 passed, 1 failed, 1 blocked of 12 (54 planned steps) |
| Browser actions | 198 (52 clicks, 20 inputs, 26 navigations, 100 snapshots) |
| Screenshots | 41 (4 explore, 36 scenario, 1 critic), 36 captioned |
| Coverage | 5 pages, 2 forms, 3 flows, 0 console errors |
| Audit | 1 findings, 1 re-verified live, 1 confirmed, 1 promoted, 0 withdrawn |
| Model calls | 204 |
| Tokens | 1,768,707 input, 10,055 output, 17,650 thinking |
| Time | 15 min |
| Stage | Calls | Input | Output | Thinking | Seconds |
|---|
| explore | 41 | 367,157 | 2,516 | 1,704 | 272 |
| plan | 1 | 3,658 | 2,290 | 2,666 | 33 |
| test | 157 | 1,383,280 | 4,505 | 8,627 | 550 |
| critique | 4 | 12,975 | 475 | 4,030 | 42 |
| report | 1 | 1,637 | 269 | 623 | 8 |
