Kotaemon

Every scenario DeepQA drove in the browser, in plan order, with the 35 screenshots it captured along the way. A passing scenario is evidence too.

1. S1

   Application Load and Tab Navigation

   5 steps, 4 screenshots

   high✓ pass

   

   

   

   

   • The application loaded initially on the Chat tab showing Conversations sidebar, file upload options, and chat interface.
   • The Files tab rendered successfully with File Upload controls, indexing options, and File List table.
   • The Resources tab rendered successfully with sub-tabs for Index Collections, LLMs, and Embeddings, and the collections grid displayed.
   • The Settings tab rendered successfully with Retrieval settings and Reasoning settings forms, chunk retrieve controls, and checkboxes.
   • The Help tab rendered successfully showing About Kotaemon, links, and the complete User Guide with images and instructions.
   • All primary navigation tabs (Chat, Files, Resources, Settings, Help) switched cleanly and rendered their respective forms and views without errors.
2. S2

   Chat Query Form Empty Submission Validation

   4 steps, 3 screenshots

   high✓ pass

   

   

   

   • Navigated to Chat tab with empty chat input box.
   • Clicked Send and pressed Enter on empty chat input textbox; no empty message bubble was created and no error was displayed.
   • The application ignores empty submissions when clicking the Send button or pressing Enter, preventing empty messages from appearing in the conversation log without throwing errors.
3. S3

   Chat Query Valid Submission

   4 steps, 3 screenshots

   high✓ pass

   

   

   

   • Navigated to the Chat interface on https://cin-model-kotaemon-demo.hf.space/ with the Chat tab active.
   • Entered query 'What is the capital of France?' into the chat input textbox.
   • Navigated to the Kotaemon Chat tab at https://cin-model-kotaemon-demo.hf.space/.
   • Observed the clear message indicating model requirements: 'This is the beginning of a new conversation. Make sure to have added a LLM by following the instructions in the Help tab.'
   • Entered query 'What is the capital of France?' into the chat textbox and clicked Send.
   • Verified the application does not crash, hang indefinitely, or display a raw traceback when submitted without configured external LLM keys.
4. S4

   Document Upload Empty Submission Validation

   4 steps, 3 screenshots

   high✓ pass

   

   

   

   • Loaded the main application interface showing the Chat tab.
   • Switched to Files tab with Drop File Here zone empty.
   • Clicked Upload and Index with no file selected; received validation notification 'No uploaded file'.
   • Navigated to the Files tab where the file drop zone was empty.
   • Clicked 'Upload and Index' without choosing any file.
   • The application gracefully rejected the empty submission and displayed an informational notification stating 'No uploaded file'.
5. S5

   Add LLM Model Missing Fields Validation

   5 steps, 4 screenshots

   medium✓ pass

   

   

   

   

   • Navigated to the Resources tab where Index Collections, LLMs, and Embeddings sub-tabs are displayed.
   • Selected the LLMs tab within Resources.
   • Opened the Add LLM form leaving the Name and Provider/vendor fields empty.
   • Clicked Add LLM with empty fields; an error notification 'Failed to create LLM : None' appeared, blocking creation.
   • Navigated to Resources > LLMs > Add tab.
   • Left the Name, Provider/Vendor, and Specification fields empty and clicked 'Add LLM'.
   • Application blocked the LLM creation and displayed an error toast 'Failed to create LLM : None'.
   • Verified on the View tab that no blank LLM record was created or added to the list.
6. S6

   Add LLM Model Missing Provider Validation

   6 steps, 4 screenshots

   medium✓ pass

   

   

   

   

   • Navigated to Resources > LLMs tab.
   • Opened Add LLM form with LLM name and LLM vendors options.
   • Clicked Add LLM with empty provider; application displayed an error 'Failed to create LLM Test Model: None' and did not save the model.
   • Navigated to Resources > LLMs > Add tab.
   • Entered 'Test Model' in the LLM name field and left the vendor/provider field unselected/empty.
   • Clicked 'Add LLM' button.
   • Observed error notification 'Failed to create LLM Test Model: None'.
   • Verified in LLMs View tab that 'Test Model' was not saved or added to the LLMs list.
7. S7

   Modify System Prompt and Save

   5 steps, 4 screenshots

   medium✓ pass

   

   

   

   

   • Opened Settings tab showing Retrieval settings and Reasoning settings tabs.
   • Navigated to Reasoning settings tab, where System Prompt field is currently set to 'This is a question answering system'.
   • Entered 'You are a helpful assistant.' into the System Prompt field.
   • Clicked 'Save changes' button; the System Prompt field maintained its value 'You are a helpful assistant.' with no errors thrown.
   • Navigated to Settings -> Reasoning settings tab.
   • Observed the default System Prompt field populated with 'This is a question answering system'.
   • Successfully edited the System Prompt textbox to 'You are a helpful assistant.'.
   • Clicked 'Save changes' button.
   • Verified the updated text 'You are a helpful assistant.' remained in the System Prompt field without errors or reverting.
8. S8

   Toggle Chat Settings Accordion

   3 steps, 2 screenshots

   low✓ pass

   

   

   • Navigated to Chat tab and located the Chat settings accordion button.
   • Clicked 'Chat settings ▼' accordion and verified that it expanded showing additional configuration options including Reasoning method and Model dropdowns.
   • The Chat tab is selected by default on load.
   • Clicking 'Chat settings ▼' successfully expands the accordion section displaying 'Reasoning method' and 'Model' configuration dropdowns without UI layout issues.
9. S9

   Toggle Information Panel Accordion

   3 steps, 4 screenshots

   low✕ fail

   

   

   

   

   • Clicked the 'Information panel ▼' accordion button on the Chat tab; no child components or citation UI/placeholder content appeared.
   • Navigated to https://cin-model-kotaemon-demo.hf.space/ and confirmed the Chat tab is active.
   • Observed other accordions such as 'Chat settings ▼' and 'Feedback ▼' expand properly with nested controls and options.
   • Clicked 'Information panel ▼' multiple times; no placeholder content, citation UI, or sub-elements were displayed under the accordion.
10. S10

    Change File Collection Scope

    3 steps, 2 screenshots

    low✓ pass

    

    

    • Initial page loaded with File Collection radio options visible under the Chat tab.
    • Selected 'Search In File(s)' radio option, which became checked and revealed the 'Files' selection input field.
    • The File Collection radio options were present on the left sidebar of the Chat interface.
    • Clicking 'Search In File(s)' successfully updated the radio button to the checked state and dynamically displayed a 'Files' selection input.
11. S11

    Regenerate Answer Without Context

    3 steps, 2 screenshots

    low✓ pass

    

    

    • Loaded the Chat tab with an empty conversation and located the Regen button.
    • Clicked the Regen button on an empty conversation, which triggered warning notifications ('No conversation selected' and 'Empty chat') rather than a backend traceback.
    • When clicking the 'Regen' button with no messages in the chat history, the application gracefully handled the action by displaying warning messages ('No conversation selected' and 'Empty chat') without throwing any backend errors or tracebacks.

Findings that survived the Critic's audit. Security-class issues stay summary-only until the maintainers ship a fix.

lowpromoted ↑uxF2 · S5

Missing field validation error exposes raw 'None' value instead of user-friendly message

Surfaced by the audit of S5, which the Tester passed, and reproduced live: I navigated to Resources > LLMs tab and opened the Add LLM form. Leaving the fields empty and clicking 'Add LLM' resulted in the error message 'Failed to create LLM : None' being displayed, reproducing the reported behaviour. The page reported 1 console error during the scenario.

Expected

The application displays a user-friendly validation error specifying which fields are required.

Actual

The application displays an unpolished error string 'Failed to create LLM : None', exposing the internal Python null representation.

Repro · 4 steps

1. Navigate to Resources > LLMs tab
2. Open the Add LLM form
3. Leave the Name and Provider/vendor fields empty
4. Click the Add LLM button

critic-M2-1.png

lowwithdrawnuxF1 · S9

Information panel accordion does not display any placeholder content or citation UI when expanded

The 'Information panel' is designed to show document citations and retrieval metadata for active chat responses. Expanding it before a query has been made naturally reveals nothing, and lacking an explicit placeholder is not a defect. The page reported 1 console error during the scenario.

Expected

The Information panel accordion should expand to reveal citation UI or placeholder content.

Actual

Clicking the 'Information panel ▼' accordion button does not reveal any child components, placeholder text, or citation UI elements in the interface.

Repro · 3 steps

1. Navigate to https://cin-model-kotaemon-demo.hf.space/
2. Ensure the Chat tab is selected
3. Click the 'Information panel ▼' accordion button

S9-1.pngS9-3.pngS9-8.pngS9-11.png

These failures came from the test environment, not from the application: a credential the sandbox does not hold, a demo nobody may write to, a resource it cannot reach. They are not counted as issues.

• S1 could not exercise this: Console logs package build error for PDF viewer component. A package build error for 'pdfjs-viewer-element' is logged to the console. The audit recorded the test environment as the cause, so it is not counted as an issue.