---
title: "Build a BPE tokenizer - in your browser, from scratch | DemystifAI"
description: "Byte Pair Encoding explained by building it: train a GPT-style tokenizer on Polish poetry and run it on the Constitution - live, in your browser, no install."
url: https://demystifai.xyz/en/blog/build-a-bpe-tokenizer/
lang: en
---

Published: July 2, 2026 3 min read + ~15 min hands-on

# Build a BPE tokenizer - in your browser, from scratch

Every LLM bill you have ever paid started with a tokenizer. And if you work with Polish text, you are quietly paying about twice as much as your English-speaking friends - for the same sentence. Today we find out why, the best way I know: by building the thing.

Reading about tokenizers is like reading about swimming. So this post *is* the pool: a [marimo](https://marimo.io) notebook compiled to WebAssembly. Real Python runs in your browser tab - no install, no server, nothing you type leaves your machine. The whole lesson lives inside it:

-   **A naive regex tokenizer** - and four real-world strings that break it.
-   **Byte Pair Encoding from scratch** - three small functions, one of them yours to write (there is an exercise with a hidden solution).
-   **Train on *Pan Tadeusz*, test on the Constitution** - watch Polish morphemes emerge from 1834 verse and compress 1997 legal text by ~1.8×.
-   **Crash a character-level tokenizer with an emoji** - and see why byte-level shrugs it off. 🚀
-   **Measure the Polish token tax** - the same sentence costs 32 tokens in Polish and 16 in English on GPT-4's tokenizer. Your API bills already know this.
-   **Four production trade-offs** - special tokens, vocab size, tokenizer versioning, pretokenization.

-   🐍 Real Python (Pyodide/WASM) - runs locally in this tab
-   🏋️ One exercise, live inputs, a train button, zero setup
-   🔒 Nothing gets uploaded anywhere

Downloads ~20 MB of Python runtime on first launch - that is why it waits for your click. Prefer a separate tab? [Open it full screen →](https://demystifai.xyz/apps/bpe-tokenizer/index.html)

My favorite part, so you know what to look for: after training, check which tokens the algorithm invented. Nobody told it what a Polish word is -  `się`, `ość` and  `prze` emerge purely from pair statistics. Then it takes those poem-learned merges and compresses the Constitution. Transfer learning, 163 years across a genre gap, in 60 lines of Python you just wrote yourself.

[The theory behind it: Tokenomics of LLMs](https://demystifai.blog/en/blog/2025-10-07-post)

Like learning by building?

This is how I run trainings and 1:1 mentoring too - on real code, with running artifacts, not slide decks. If your team is adopting AI tooling and you want it to actually stick, let's talk.

[Training for dev teams →](https://demystifai.xyz/en/for-teams/) [Working solo? Check 1:1 mentoring →](https://demystifai.xyz/en/mentoring-premium/)

## What now?

If you enjoyed learning by poking a running artifact instead of nodding at slides - that is not an accident, that is the method:

[Training for software houses and dev teams → Hands-on AI-SDLC workshops for 20-100+ devs - artifacts included, hype not.](https://demystifai.xyz/en/for-teams/) [1:1 mentoring for founders → 8 weeks on your code and your business. First session risk-free.](https://demystifai.xyz/en/mentoring-premium/) [AI consulting - implementations for companies → Have a concrete project on a deadline? I will ship it with you.](https://demystifai.xyz/en/consulting/)

[Book a 30-minute call →](https://calendar.app.google/LM1HVxiH2dbFdpjj6)

Questions? Drop me a line at tomasz@gusciora.pl or grab a 30-minute call [here](https://calendar.app.google/LM1HVxiH2dbFdpjj6). Corpus credits: *Pan Tadeusz* via [Wolne Lektury](https://wolnelektury.pl/katalog/lektura/pan-tadeusz/), Konstytucja RP via [Wikisource](https://pl.wikisource.org/wiki/Konstytucja_Rzeczypospolitej_Polskiej) - both public domain. See you on DemystifAI!

## Want more solid AI material, minus the hype?

I'm building a contact list for extra materials on AI and Claude Code - battle-tested, no fluff. Leave your email and I'll reach out when I have something genuinely worth your time.

No spam. By subscribing you agree to receive the newsletter from DemystifAI Tomasz Guściora. Unsubscribe anytime. Details in the [Privacy Policy](https://demystifai.xyz/en/privacy-policy/).