Tokenizer Playground

What this page will cover

• Interactive tokenizer component (TODO)
• Things to try: English prose, code, a base64 blob, a hex string, non-English text, emoji
• Observe: token count vs. character count

Security lens

Unicode tricks (homoglyphs, invisible characters) change tokenization, which is why string-matching filters on model input are fragile.