2:13:35Long-form

Let's build the GPT Tokenizer

United StatesPublished February 20, 2024

Views 1.17M  |  Likes 28.6K  |  Comments 1.20K
Views#645,198Global
Views#149,899United States
Live count

Channel

Andrej KarpathyAndrej Karpathy
1.65M subs•17 videos•Sep 2013

At a glance

Views

1,171,190

Likes

28,679

Comments

1,202

Performance vs. channel

Outlier score

Engagement rate

Revenue estimate

Details →

Total (all-time): $820 – $2.34K

Based on estimated $0.7–$2 CPM.

About

The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement tw…

Published
February 20, 2024
Made for kids
No

More features coming

We're working on new analytics and tools. Stay tuned.