No application defects were confirmed during testing across read-only inspection and monitoring workflows.
Testing exercised twelve scenarios evaluating project navigation, trace waterfalls, span inspection, telemetry metrics, conversation sessions, and configuration views. All read-only visualization and browsing capabilities behaved as expected.
Two scenarios attempting write actions—project creation and retention policy updates—failed due to read-only restrictions on this public demo deployment. The audit reviewed the two corresponding findings (F1 and F2) and withdrew both as test environment limitations rather than application defects.
Because the demo environment prohibits mutations, write-path operations such as project creation, data updates, and duplicate name validation could not be fully exercised. Testing on a writable environment will be necessary to validate data persistence and mutation flows before release decisions.
| Metric | Count |
|---|---|
| Scenarios executed | 12 |
| Passed | 10 |
| Failed | 2 |
| Blocked | 0 |
| Findings raised | 2 |
| Issues after the audit | 0 |
| Withdrawn by the audit | 2 |
| Critical / high / medium / low | 0 / 0 / 0 / 0 |
Target: https://phoenix-demo.arize.com · Testing level: deep_feature · Stack: unknown
No issues survived the audit.
These failures came from the test environment, not from the application: a credential the sandbox does not hold, a demo nobody may write to, a resource it cannot reach. They are not counted as issues. They record what this run could not exercise.
- Default Project Retention Policy update fails with error and does not persist (S12, high): The failure to update the retention policy and its reversion on reload are expected behaviors on a read-only demo instance, not application defects. The audit recorded the test environment as the cause, so this is a limit of the run rather than a defect of the application.
- Project creation fails with 'Application is read-only' error (S2, medium): The 'Application is read-only' error indicates the test is running against a restricted public demo, making this an environment limit rather than an application defect. The audit recorded the test environment as the cause, so this is a limit of the run rather than a defect of the application.
- S6 could not exercise this: Duplicate project name validation blocked by read-only environment. The submission was blocked by a global read-only environment restriction, preventing the application's duplicate name logic from being exercised. The audit recorded the test environment as the cause, so…
| Scenario | Priority | Result | Issues |
|---|---|---|---|
| S1 Validate empty project name on creation | high | pass | none |
| S2 Create a new project successfully | high | fail | none |
| S3 Search and filter projects list | high | pass | none |
| S4 Load project spans data | high | pass | none |
| S5 Inspect trace waterfall and span details | high | pass | none |
| S6 Submit duplicate project name | medium | pass | none |
| S7 Toggle between grid and table layouts | medium | pass | none |
| S8 View project end-to-end traces | medium | pass | none |
| S9 View project conversation sessions | medium | pass | none |
| S10 View project telemetry metrics | medium | pass | none |
| S11 View project configuration | medium | pass | none |
| S12 Update default project retention policy | low | fail | none |
The Critic reviewed 2 findings and re-verified 0 of them live in the browser, replaying the reported steps on a fresh page.
- The target application is a read-only public demo, causing all scenarios that attempt write operations to hit environment restrictions rather than application behavior.
In scope: Projects listing, view toggles, and search functionality; Project creation lifecycle and form validation; Deep trace analysis via Spans, Traces, Sessions, Metrics, and Config tabs; Global settings related to project retention policies.
Not covered: Datasets and experiment management (outside deep feature focus); Evaluators, Prompts, Playground, and Chat (outside deep feature focus); REST and GraphQL API testing (UI-only testing scope).
- The demo instance allows write actions such as project creation and settings updates; if strictly read-only, write scenarios will fail gracefully.
- Project deletion and edit affordances were not mapped and are assumed absent or unsupported by the current UI.
- Pre-populated demo data (e.g., 'demo_agents') is consistently available for read-only validation of spans, traces, and metrics.
| Metric | Value |
|---|---|
| Scenarios | 10 passed, 2 failed, 0 blocked of 12 (38 planned steps) |
| Browser actions | 156 (36 clicks, 8 inputs, 31 navigations, 81 snapshots) |
| Screenshots | 39 (4 explore, 35 scenario, 0 critic), 35 captioned |
| Coverage | 15 pages, 4 forms, 3 flows, 0 console errors |
| Audit | 2 findings, 0 re-verified live, 0 confirmed, 0 promoted, 2 withdrawn |
| Model calls | 161 |
| Tokens | 1,233,898 input, 9,292 output, 16,073 thinking |
| Time | 18 min |
| Stage | Calls | Input | Output | Thinking | Seconds |
|---|---|---|---|---|---|
| explore | 31 | 238,669 | 2,682 | 884 | 122 |
| plan | 1 | 4,576 | 1,971 | 4,612 | 50 |
| test | 127 | 984,058 | 4,015 | 7,038 | 866 |
| critique | 1 | 4,931 | 438 | 3,027 | 29 |
| report | 1 | 1,664 | 186 | 512 | 7 |