Generative AI
Tokenization
How text becomes numbers. Byte-pair encoding, why token counts matter and how to count them, with a live tokenizer.
- Intermediate
- About 4 hours
- 1 modules
- 5 lessons
- Free
This course is not released yet. We publish courses one at a time, each one complete. The full syllabus is below.
What you will learn
- TokensWhat Is a Token?, Byte-Pair Encoding, Special Tokens and more
Course syllabus
Not released yet · full syllabus below
- 1
Tokens
5 lessons- 1What Is a Token?Soon
- 2Byte-Pair EncodingSoon
- 3Special TokensSoon
- 4Counting TokensSoon
- 5Tokenisation Across LanguagesSoon