QA report: external/chat.webllm.ai at hosted
The application functions well overall, but the conversation settings modal mutates topic titles immediately without requiring confirmation.
A deep feature evaluation tested twelve scenarios covering core interface navigation, settings management, prompt library interactions, export functionality, and state persistence. Eleven scenarios passed, with one failing due to a functional defect in modal state handling.
The single failure occurred during conversation settings editing (F1). Typing into the conversation topic input field binds reactively and updates the topic across the header and sidebar in real time, persisting the modified name even when the modal is closed or cancelled without clicking Confirm.
Due to the test environment lacking WebGPU support, actual in-browser model initialization and full LLM inference could not be exercised, leaving generation performance unverified.
Run summary
| Metric | Count |
|---|
| Scenarios executed | 12 |
| Passed | 11 |
| Failed | 1 |
| Blocked | 0 |
| Findings raised | 1 |
| Issues after the audit | 1 |
| Withdrawn by the audit | 0 |
| Critical / high / medium / low | 0 / 0 / 1 / 0 |
Target: https://chat.webllm.ai/ · Testing level: deep_feature · Stack: unknown
Issues
Medium severity
F1 · Conversation Settings modal immediately mutates topic without clicking Confirm
Severity: medium · Type: functional · Verdict: confirmed · Scenario: S9
The observations confirm that modifying the topic input instantly updates the application state, and dismissing the modal via Escape fails to revert the unconfirmed changes. The page reported 1 console error during the scenario.
Expected: The conversation topic should remain unchanged until the user explicitly saves changes by clicking Confirm, and should revert if the modal is dismissed.
Actual: The conversation topic input field binds reactively to the application state immediately upon typing, updating the topic displayed in the header and sidebar in real time, and persists the changes even when the modal is dismissed without clicking Confirm.
Steps to reproduce:
- Navigate to https://chat.webllm.ai/
- Click the edit button in the chat header to open the Conversation Settings modal
- Type 'Should Not Save' into the Topic field
- Dismiss the modal without clicking Confirm (by pressing Escape)
- Observe the conversation topic in the header and sidebar
Evidence: screenshots/S9-5.png, screenshots/S9-7.png, screenshots/S9-10.png
Environment limitations
These failures came from the test environment, not from the application: a credential the sandbox does not hold, a demo nobody may write to, a resource it cannot reach. They are not counted as issues. They record what this run could not exercise.
- S2 could not exercise this: Model initialization fails due to missing WebGPU support in sandbox. The model fails to initialize because the test environment lacks WebGPU support. The audit recorded the test environment as the cause, so it is not counted as an issue.
Scenario results
| Scenario | Priority | Result | Issues |
|---|
| S1 Application Load and Primary Interface Availability | high | pass | none |
| S2 Graceful Hardware Failure on Message Send | high | pass | none |
| S3 Update Conversation Topic | high | pass | none |
| S4 Persist Conversation Settings Across Reloads | medium | pass | none |
| S5 Empty Message Validation | medium | pass | none |
| S6 Toggle Hide Context Prompts | medium | pass | none |
| S7 Export Conversation as Markdown | medium | pass | none |
| S8 Navigate and Close Prompt Library | medium | pass | none |
| S9 Cancel Conversation Settings Edits | low | fail | F1 |
| S10 Prompt Library Export Function | low | pass | none |
| S11 Access Global Settings Modal | low | pass | none |
| S12 Trigger Share Conversation Action | low | pass | none |
The audit
The Critic reviewed 1 finding and re-verified 0 of them live in the browser, replaying the reported steps on a fresh page.
- The run successfully navigated the UI but was unable to test actual LLM inference due to the sandbox lacking WebGPU support.
What to fix first
- F1: Buffer conversation settings input locally within the modal so changes only apply and persist to application state when confirmed.
Coverage and caveats
In scope: Chat interface and message input handling; Graceful error handling for missing WebGPU support; Conversation settings modification (Topic, Context Prompts); Persistence of conversation settings across reloads; Export and Share interface interactions; Prompt library and global settings navigation.
Not covered: Successful LLM message inference (requires hardware WebGPU support unavailable in the headless test environment); Model downloading and caching mechanics (dependent on successful GPU initialization).
- The test runner operates in a non-GPU environment, so inference will predictably fail with a 'Unable to find a compatible GPU' error.
- The application initializes a default conversation upon first load, making the edit controls immediately available.
- Conversations are stored in LocalStorage or IndexedDB, meaning changes should persist across page reloads in the same session.
By the numbers
| Metric | Value |
|---|
| Scenarios | 11 passed, 1 failed, 0 blocked of 12 (39 planned steps) |
| Browser actions | 164 (41 clicks, 11 inputs, 20 navigations, 92 snapshots) |
| Screenshots | 40 (4 explore, 36 scenario, 0 critic), 36 captioned |
| Coverage | 4 pages, 2 forms, 4 flows, 3 console errors |
| Audit | 1 findings, 0 re-verified live, 1 confirmed, 0 promoted, 0 withdrawn |
| Model calls | 166 |
| Tokens | 758,440 input, 7,899 output, 13,044 thinking |
| Time | 7 min |
| Stage | Calls | Input | Output | Thinking | Seconds |
|---|
| explore | 32 | 138,204 | 1,770 | 2,002 | 101 |
| plan | 1 | 3,835 | 1,989 | 3,506 | 40 |
| test | 131 | 609,658 | 3,575 | 5,873 | 278 |
| critique | 1 | 5,204 | 356 | 1,360 | 13 |
| report | 1 | 1,539 | 209 | 303 | 5 |
