What Are Tokens?

What this page will cover

• Tokens as sub-word chunks mapped to integer IDs
• Analogy: a lexer, or a compression dictionary (byte-pair encoding)
• Why hello, hello, and HELLO are different tokens
• Tokens as the unit of cost and context length