QA report: external/demo.sourcebot.dev at hosted
Core search and repository browsing workflows are functional, with one low-severity defect causing whitespace-only searches to trigger a 400 error.
Testing covered 12 scenarios across core features including cross-repository code search, file navigation, repository filtering, and chat UI settings. Two scenarios were blocked by environment constraints: asking AI queries required an authenticated account, and switching AI models could not be tested because only one model was available on the hosted instance.
A single low-severity functional defect was confirmed during the audit. Submitting a search query consisting entirely of whitespace passes client validation and triggers an unhandled 400 Bad Request error from the API. Two other reported issues regarding the branch selector and chat onboarding walkthrough were audited and withdrawn.
The application remains stable across its primary browsing and navigation paths. Resolving the client-side search input trimming will ensure clean error-free handling for edge-case query submissions.
Run summary
| Metric | Count |
|---|
| Scenarios executed | 12 |
| Passed | 8 |
| Failed | 2 |
| Blocked | 2 |
| Findings raised | 2 |
| Issues after the audit | 1 |
| Withdrawn by the audit | 2 |
| Critical / high / medium / low | 0 / 0 / 0 / 1 |
Target: https://demo.sourcebot.dev · Testing level: deep_feature · Stack: unknown
Issues
Low severity
F3 · Whitespace-only search queries trigger a 400 Bad Request error
Severity: low · Type: functional · Verdict: promoted · Scenario: S5
Surfaced by the audit of S5, which the Tester passed, and reproduced live: I entered a whitespace into the search box and pressed Enter on the home page. The application navigated to /search?query=%20 and displayed an error message: "❌ Search failed. Reason: Failed to parse query " " with message: No parse at 1". This confirms the whitespace query is submitted and fails instead of being handled client-side. The page reported 2 console errors during the scenario.
Expected: Whitespace queries should be prevented or handled client-side without submitting invalid requests that cause a 400 server error.
Actual: The application submits the whitespace query to the API, resulting in a 400 Bad Request and logging an error to the console.
Steps to reproduce:
- Navigate to the home page.
- Enter whitespace into the search input and submit.
Evidence: screenshots/critic-M1-1.png
Withdrawn findings
The Critic re-examined these claims and found the evidence did not support them. They are kept here rather than deleted.
- Repository branch selector is unresponsive or inaccessible on repository tree view (S8, medium): The claim is based on what the Tester assumed the page ought to do (have a branch selector) rather than an observation of a specific broken control.
- Chat onboarding walkthrough dialog or tutorial overlay does not render on initial visit (S10, low): Scenario S4 directly contradicts this finding, as it successfully observed and dismissed the Ask Sourcebot tutorial dialog earlier in the run.
Scenario results
| Scenario | Priority | Result | Issues |
|---|
| S1 Search code across repositories | high | pass | none |
| S2 Filter indexed repositories | high | pass | none |
| S3 Browse repository files | high | pass | none |
| S4 Ask AI a question | high | blocked (needs an account, out of scope for a hosted run) | none |
| S5 Prevent empty search queries | medium | pass | F3 |
| S6 Prevent empty chat prompts | medium | pass | none |
| S7 Select repository scope for chat | medium | pass | none |
| S8 Open repository branch selector | medium | fail | none |
| S9 Change AI model in chat | low | blocked (Only one model (claude-opus-4-8) is available in the model selector on this hosted instance, preventing selection of an alternate model.) | none |
| S10 View chat onboarding walkthrough | low | fail | none |
| S11 Open What's New dialog | low | pass | none |
| S12 Use global navigation links | low | pass | none |
The audit
The Critic reviewed 2 findings and re-verified 1 of them live in the browser, replaying the reported steps on a fresh page.
- S10's failure resulted from state persisting after S4 dismissed the tutorial dialog, which the Tester incorrectly logged as a defect.
- S8 recorded a failure because the Tester could not find a feature they assumed should be present, rather than testing what was actually rendered.
What to fix first
- Prevent whitespace-only search query submissions on the client side or handle them gracefully to avoid API 400 Bad Request errors (F3).
Coverage and caveats
In scope: Code search querying and results display; Repository list filtering and file tree browsing; AI chat assistant interactions, including prompt validation; AI chat configuration (repository scope and model selection).
Not covered: User authentication and sign-in (needs an account, out of scope for a hosted run).
- The demo repositories (e.g., github.com/adobe/skills) are consistently available and indexed in the hosted environment.
- Could not cover file attachment functionality in the chat due to the 12-scenario cap.
- S4 could not be executed: needs an account, out of scope for a hosted run.
- S9 could not be executed: Only one model (claude-opus-4-8) is available in the model selector on this hosted instance, preventing selection of an alternate model..
By the numbers
| Metric | Value |
|---|
| Scenarios | 8 passed, 2 failed, 2 blocked of 12 (38 planned steps) |
| Browser actions | 188 (43 clicks, 20 inputs, 21 navigations, 104 snapshots) |
| Screenshots | 35 (4 explore, 30 scenario, 1 critic), 30 captioned |
| Coverage | 6 pages, 3 forms, 3 flows, 0 console errors |
| Audit | 2 findings, 1 re-verified live, 0 confirmed, 1 promoted, 2 withdrawn |
| Model calls | 173 |
| Tokens | 1,389,733 input, 8,621 output, 18,253 thinking |
| Time | 9 min |
| Stage | Calls | Input | Output | Thinking | Seconds |
|---|
| explore | 30 | 296,362 | 1,845 | 1,236 | 82 |
| plan | 1 | 3,733 | 1,844 | 4,129 | 44 |
| test | 136 | 1,073,374 | 3,983 | 8,379 | 376 |
| critique | 5 | 14,512 | 696 | 3,991 | 45 |
| report | 1 | 1,752 | 253 | 518 | 8 |
