QA report: okisdev/ChatChat at 1267428
No confirmed defects were found across the tested chat, session management, and configuration features.
Twelve scenarios were executed to evaluate core interface and chat workflows, including session creation and deletion, state persistence across reloads, whitespace validation, system prompt configuration, export settings, and interface language toggling.
Ten scenarios passed cleanly. A single finding raised during empty message validation was withdrawn after audit verification determined that leftover state from an earlier scenario caused a false failure rather than an application defect. Additionally, testing the model provider switcher was blocked before completion due to navigation budget limits in nested provider menus.
Because no confirmed application defects were identified, core chat operations remain stable. Complete verification of model provider switching will require coverage in a subsequent run.
Run summary
| Metric | Count |
|---|
| Scenarios executed | 12 |
| Passed | 10 |
| Failed | 1 |
| Blocked | 1 |
| Findings raised | 1 |
| Issues after the audit | 0 |
| Withdrawn by the audit | 1 |
| Critical / high / medium / low | 0 / 0 / 0 / 0 |
Target: http://[::1]:3000 · Testing level: deep_feature · Stack: node / pnpm (next)
Issues
No issues survived the audit.
Withdrawn findings
The Critic re-examined these claims and found the evidence did not support them. They are kept here rather than deleted.
- Submitting empty message input creates a new session in sidebar (S3, medium): The 'Hello, world!' chat session was created in scenario S2 and persisted in local storage across tests. The tester mistakenly attributed this pre-existing leftover state to the empty message submission in S3. The page reported 7 console errors during the scenario.
Scenario results
| Scenario | Priority | Result | Issues |
|---|
| S1 Main Interface Load | high | pass | none |
| S2 Create New Chat Session | high | pass | none |
| S3 Empty Message Validation | high | fail | none |
| S4 Whitespace Message Validation | high | pass | none |
| S5 Chat Session Persistence on Reload | high | pass | none |
| S6 Navigate to Specific Chat Session | high | pass | none |
| S7 Switch LLM Model Provider | medium | blocked (Ran out of tool call budget while attempting to select an unconfigured Anthropic model item from the nested provider menu.) | none |
| S8 Configure System Prompt | medium | pass | none |
| S9 Delete Chat Session | medium | pass | none |
| S10 Toggle Interface Language | low | pass | none |
| S11 Open Export / Conversation Settings | low | pass | none |
| S12 View Version Information | low | pass | none |
The audit
The Critic reviewed 1 finding and re-verified 2 of them live in the browser, replaying the reported steps on a fresh page.
- The test harness fails to isolate local storage state between scenarios, causing session state bleed from S2 to S3 and leading to a false functional finding.
- A possible defect in S2 was called an environment limitation, and nothing that scenario recorded names an environment cause, so it was audited as a candidate.
- A possible defect in S1 ("React hydration error on initial page load") was not promoted: the live replay came back not-reproduced.
- A possible defect in S2 ("500 Internal Server Error when submitting a chat message") was not promoted: the live replay came back inconclusive.
Coverage and caveats
In scope: Chat session lifecycle (create, view, delete); Local state persistence; Model and provider selection; Empty and whitespace input validation; UI configuration (language, theme, system prompt).
Not covered: External LLM response validation (requires API keys, out of scope for pure UI testing); API key configuration (surface not fully mapped in the provided AppMap).
- The application renders the user's message in the UI immediately upon sending, regardless of whether the external LLM API request succeeds or fails due to missing keys.
- The theme switcher is accessible via a button near the language selector.
- Hydration errors in the console do not fundamentally break the primary React component tree for the chat interface.
- S7 could not be executed: Ran out of tool call budget while attempting to select an unconfigured Anthropic model item from the nested provider menu..
By the numbers
| Metric | Value |
|---|
| Scenarios | 10 passed, 1 failed, 1 blocked of 12 (36 planned steps) |
| Browser actions | 232 (54 clicks, 19 inputs, 22 navigations, 137 snapshots) |
| Screenshots | 46 (4 explore, 41 scenario, 1 critic), 41 captioned |
| Coverage | 2 pages, 1 forms, 3 flows, 10 console errors |
| Audit | 1 findings, 2 re-verified live, 0 confirmed, 0 promoted, 1 withdrawn |
| Model calls | 202 |
| Tokens | 756,954 input, 8,496 output, 17,110 thinking |
| Time | 10 min |
| Stage | Calls | Input | Output | Thinking | Seconds |
|---|
| explore | 41 | 170,824 | 1,723 | 1,365 | 163 |
| plan | 1 | 3,563 | 1,923 | 2,304 | 31 |
| test | 146 | 544,062 | 3,928 | 8,189 | 331 |
| critique | 13 | 36,894 | 746 | 4,588 | 64 |
| report | 1 | 1,611 | 176 | 664 | 6 |
