Tiktokenizer

Every scenario DeepQA drove in the browser, in plan order, with the 34 screenshots it captured along the way. A passing scenario is evidence too.

1. S1

   Token count updates dynamically in chat mode

   3 steps, 2 screenshots

   high✓ pass

   

   

   • Initial page loaded with default system message and empty user message, token [redacted] is 16.
   • Entered 'Testing the chat tokenization' in the user message textarea, token [redacted] updated to 21 and token [redacted] ('Testing', 'the', 'chat', 'token', 'ization') appeared.
   • The initial chat view rendered a default system prompt with a token [redacted] of 16.
   • Typing 'Testing the chat tokenization' into the content textarea dynamically increased the token [redacted] from 16 to 21.
   • The colored token [redacted] accurately reflected the entered text split into sub-tokens ('Testing', 'the', 'chat', 'token', 'ization') along with their corresponding IDs.
2. S2

   Token count updates dynamically in raw encoder mode

   3 steps, 2 screenshots

   high✓ pass

   

   

   • Navigated to Tiktokenizer with cl100k_base model loaded in raw encoder mode with initial token [redacted] 0.
   • Typed 'Testing the raw encoder tokenization' and observed token [redacted] updated to 6 with colored token [redacted] and token IDs rendered properly.
   • The raw encoder mode accurately parsed the input text into 6 tokens and rendered both text chips and token IDs dynamically.
3. S3

   Add a new message to the chat

   3 steps, 3 screenshots

   high✓ pass

   

   

   

   • Loaded Tiktokenizer home page with initial token [redacted] of 16.
   • Clicked 'Add message' and observed a new message field added to the chat.
   • Typed 'A new user message' into the added message textarea; token [redacted] increased from 16 to 24 and tokens were updated.
   • Navigated to Tiktokenizer home page with initial token [redacted] of 16.
   • Clicked 'Add message' button, successfully creating a new message block.
   • Entered 'A new user message' into the new Content textarea, which updated the token [redacted] from 16 to 24 and displayed the individual tokens for the new message.
4. S4

   Delete a message from the chat

   4 steps, 3 screenshots

   high✓ pass

   

   

   

   • Initial state has two message blocks and a total token [redacted] of 16.
   • Added message and text, increasing token [redacted] from 16 to 22.
   • After deleting the second message, the message block was removed and token [redacted] decreased back from 22 to 16.
   • Navigated to https://tiktokenizer.vercel.app/ with default 2 messages and token [redacted] 16.
   • Added a third message and populated the second message with text 'Hello world', which increased token [redacted] to 22.
   • Clicked the delete message button on the second message block; the message field disappeared and token [redacted] decreased back to 16.
5. S5

   Switch between chat and raw encoder interfaces

   3 steps, 2 screenshots

   high✓ pass

   

   

   • Selected cl100k_base encoder: multi-turn chat interface disappeared and was replaced by the single raw encoder input textarea.
   • Navigated to https://tiktokenizer.vercel.app/ which initially loaded in chat mode with multi-turn message textareas and role selectors.
   • Opened the Model/Encoder combobox dialog, searched for and selected 'cl100k_base'.
   • Observed that the multi-turn chat interface disappeared and was replaced by the single raw encoder input textarea.
   • Verified entering text into the raw encoder textarea tokenizes properly (token [redacted] updated to 7 for 'Hello world from cl100k_base').
6. S6

   Change message role in chat mode

   3 steps, 3 screenshots

   medium✓ pass

   

   

   

   • Loaded Tiktokenizer initial chat mode showing messages with role selectors.
   • Changed the role selector of the second message to 'system', and observed the serialized tokens update from 'user' (token ID 1428) to 'system' (token ID 17360).
   • Navigated to https://tiktokenizer.vercel.app/ in chat mode.
   • Opened the role combobox for the second chat message and selected 'System'.
   • Observed the serialized prompt and tokenized representation update immediately, showing the 'system' token (token ID 17360) in place of 'user' (token ID 1428).
7. S7

   Search for a model in the dialog

   4 steps, 3 screenshots

   medium✓ pass

   

   

   

   • Loaded the Tiktokenizer home page.
   • Opened the model / encoder selector dialog showing the search input and available models.
   • Typed 'gpt2' into the search input and observed that the model list filtered to only display the matching 'gpt2' option.
   • The model/encoder combobox dialog opened properly on click and the search input filtered the options list to display matching model 'gpt2' when 'gpt2' was typed.
8. S8

   Toggle whitespace visualization in chat mode

   4 steps, 4 screenshots

   medium✓ pass

   

   

   

   

   • Entered 'Hello World' into the message content textarea and observed tokens generated for Hello and World.
   • Toggling the 'Show whitespace' checkbox converts spaces in token [redacted] into visual dot indicators (⋅).
   • Confirmed that tokens for 'Hello World' visually represented consecutive whitespace tokens as '⋅⋅⋅' and '⋅World' instead of standard spaces.
9. S9

   Toggle whitespace visualization in raw encoder mode

   4 steps, 3 screenshots

   medium✓ pass

   

   

   

   • Navigated to tiktokenizer with cl100k_base model selected.
   • Entered 'Space test ' into the raw encoder input and observed 3 tokens with default whitespace display.
   • Clicked the 'Show whitespace' checkbox, and spaces in token [redacted] were visually rendered as middle dots ('⋅test', '⋅⋅⋅⋅').
   • The raw encoder tokenizes 'Space test ' into 3 tokens (10115, 1296, 257).
   • Toggling 'Show whitespace' dynamically updates the token [redacted] to render space characters visually as '⋅', clearly differentiating whitespace in '⋅test' and '⋅⋅⋅⋅'.
10. S10

    Load model selection directly from URL parameter

    2 steps, 3 screenshots

    medium✓ pass

    

    

    

    • Navigated to https://tiktokenizer.vercel.app/?model=gpt2 and confirmed the gpt2 model/tokenizer loaded successfully with gpt2 tokenization (15496, 995 for 'Hello world').
    • Navigating to https://tiktokenizer.vercel.app/?model=gpt2 successfully initialized the model directly to gpt2.
    • Entering 'Hello world' verified that the gpt2 encoder was active, outputting token [redacted] 2 with gpt2 token IDs [15496, 995].
11. S11

    Handle empty message content gracefully

    3 steps, 3 screenshots

    low✓ pass

    

    

    

    • Initial page loaded with default system message 'You are a helpful assistant' and token [redacted] 16.
    • Cleared the system message content; the token [redacted] updated cleanly from 16 to baseline structural token [redacted] 11.
    • Entered new text into the user message textarea and verified token [redacted] updated to 13 as expected.
    • Empty message content is handled gracefully by recalculating structural baseline tokens (token [redacted] 11) without error or NaN.
    • The visualizer remains fully interactive and responsive to subsequent text inputs.
12. S12

    Handle empty raw encoder input gracefully

    3 steps, 3 screenshots

    low✓ pass

    

    

    

    • Navigated to https://tiktokenizer.vercel.app/?model=cl100k_base where the raw encoder loaded with an empty textbox and token [redacted] 0.
    • Typed 'Hello world this is a test' into the Raw encoder input, resulting in Token [redacted] 6 and 6 token [redacted]
    • Cleared all text in the Raw encoder textarea; the token [redacted] immediately updated to 0 and all token [redacted] disappeared.
    • Navigated to https://tiktokenizer.vercel.app/?model=cl100k_base with raw encoder mode.
    • Entered text into the textarea and verified token [redacted] and token [redacted] appeared.
    • Cleared all text from the raw encoder textarea and observed token [redacted] display exactly 0 with no token [redacted] rendered.