huggingface-tokenizers

Text has to be chopped into small pieces and turned into numbers before a model can read it; this does the chopping at about a gigabyte every twenty seconds, and can learn a set of pieces tailored to your own kind of writing.

WHEN TO USE

SCOPE

USE FOR

Train a custom BPE or WordPiece vocabulary on your own text corpus from scratch.
Chop a huge pile of text fast enough to keep a production system fed.
Map a model's predicted token back to its exact character span in the source text.
Wrap a custom-trained tokenizer so it works as a normal HuggingFace transformers tokenizer.

NOT FOR

For a language-independent tokenizer like the one T5 or ALBERT use, use SentencePiece instead.
For OpenAI's GPT-model BPE tokenizer, use tiktoken instead.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (12)

HuggingFace Tokenizers - Fast Tokenization for NLP

Fast, production-ready tokenizers with Rust performance and Python ease-of-use.

When to use HuggingFace Tokenizers

Use HuggingFace Tokenizers when:

Need extremely fast tokenization (<20s per GB of text)
Training custom tokenizers from scratch
Want alignment tracking (token → original text position)
Building production NLP pipelines
Need to tokenize large corpora efficiently

Performance:

Speed: <20 seconds to tokenize 1GB on CPU
Implementation: Rust core with Python/Node.js bindings
Efficiency: 10-100× faster than pure Python implementations

Use alternatives instead:

SentencePiece: Language-independent, used by T5/ALBERT
tiktoken: OpenAI's BPE tokenizer for GPT models
transformers AutoTokenizer: Loading pretrained only (uses this library internally)

Quick start

Installation

# Install tokenizers
pip install tokenizers

# With transformers integration
pip install tokenizers transformers

Load pretrained tokenizer

from tokenizers import Tokenizer

# Load from HuggingFace Hub
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

# Encode text
output = tokenizer.encode("Hello, how are you?")
print(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']
print(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]

# Decode back
text = tokenizer.decode(output.ids)
print(text)  # "hello, how are you?"

Train custom BPE tokenizer

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

# Initialize tokenizer with BPE model
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

# Configure trainer
trainer = BpeTrainer(
    vocab_size=30000,
    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
    min_frequency=2
)

# Train on files
files = ["train.txt", "validation.txt"]
tokenizer.train(files, trainer)

# Save
tokenizer.save("my-tokenizer.json")

Training time: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB

Batch encoding with padding

# Enable padding
tokenizer.enable_padding(pad_id=3, pad_token="[PAD]")

# Encode batch
texts = ["Hello world", "This is a longer sentence"]
encodings = tokenizer.encode_batch(texts)

for encoding in encodings:
    print(encoding.ids)
# [101, 7592, 2088, 102, 3, 3, 3]
# [101, 2023, 2003, 1037, 2936, 6251, 102]

Tokenization algorithms

BPE (Byte-Pair Encoding)

How it works:

Start with character-level vocabulary
Find most frequent character pair
Merge into new token, add to vocabulary
Repeat until vocabulary size reached

Used by: GPT-2, GPT-3, RoBERTa, BART, DeBERTa

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel

tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>"))
tokenizer.pre_tokenizer = ByteLevel()

trainer = BpeTrainer(
    vocab_size=50257,
    special_tokens=["<|endoftext|>"],
    min_frequency=2
)

tokenizer.train(files=["data.txt"], trainer=trainer)

Advantages:

Handles OOV words well (breaks into subwords)
Flexible vocabulary size
Good for morphologically rich languages

Trade-offs:

Tokenization depends on merge order
May split common words unexpectedly

WordPiece

How it works:

Start with character vocabulary
Score merge pairs: frequency(pair) / (frequency(first) × frequency(second))
Merge highest scoring pair
Repeat until vocabulary size reached

Used by: BERT, DistilBERT, MobileBERT

from tokenizers import Tokenizer
from tokenizers.models import WordPiece
from tokenizers.trainers import WordPieceTrainer
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.normalizers import BertNormalizer

tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))
tokenizer.normalizer = BertNormalizer(lowercase=True)
tokenizer.pre_tokenizer = Whitespace()

trainer = WordPieceTrainer(
    vocab_size=30522,
    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
    continuing_subword_prefix="##"
)

tokenizer.train(files=["corpus.txt"], trainer=trainer)

Advantages:

Prioritizes meaningful merges (high score = semantically related)
Used successfully in BERT (state-of-the-art results)

Trade-offs:

Unknown words become [UNK] if no subword match
Saves vocabulary, not merge rules (larger files)

Unigram

How it works:

Start with large vocabulary (all substrings)
Compute loss for corpus with current vocabulary
Remove tokens with minimal impact on loss
Repeat until vocabulary size reached

Used by: ALBERT, T5, mBART, XLNet (via SentencePiece)

from tokenizers import Tokenizer
from tokenizers.models import Unigram
from tokenizers.trainers import UnigramTrainer

tokenizer = Tokenizer(Unigram())

trainer = UnigramTrainer(
    vocab_size=8000,
    special_tokens=["<unk>", "<s>", "</s>"],
    unk_token="<unk>"
)

tokenizer.train(files=["data.txt"], trainer=trainer)

Advantages:

Probabilistic (finds most likely tokenization)
Works well for languages without word boundaries
Handles diverse linguistic contexts

Trade-offs:

Computationally expensive to train
More hyperparameters to tune

Tokenization pipeline

Complete pipeline: Normalization → Pre-tokenization → Model → Post-processing

Normalization

Clean and standardize text:

from tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence

tokenizer.normalizer = Sequence([
    NFD(),           # Unicode normalization (decompose)
    Lowercase(),     # Convert to lowercase
    StripAccents()   # Remove accents
])

# Input: "Héllo WORLD"
# After normalization: "hello world"

Common normalizers:

NFD, NFC, NFKD, NFKC - Unicode normalization forms
Lowercase() - Convert to lowercase
StripAccents() - Remove accents (é → e)
Strip() - Remove whitespace
Replace(pattern, content) - Regex replacement

Pre-tokenization

Split text into word-like units:

from tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel

# Split on whitespace and punctuation
tokenizer.pre_tokenizer = Sequence([
    Whitespace(),
    Punctuation()
])

# Input: "Hello, world!"
# After pre-tokenization: ["Hello", ",", "world", "!"]

Common pre-tokenizers:

Whitespace() - Split on spaces, tabs, newlines
ByteLevel() - GPT-2 style byte-level splitting
Punctuation() - Isolate punctuation
Digits(individual_digits=True) - Split digits individually
Metaspace() - Replace spaces with ▁ (SentencePiece style)

Post-processing

Add special tokens for model input:

from tokenizers.processors import TemplateProcessing

# BERT-style: [CLS] sentence [SEP]
tokenizer.post_processor = TemplateProcessing(
    single="[CLS] $A [SEP]",
    pair="[CLS] $A [SEP] $B [SEP]",
    special_tokens=[
        ("[CLS]", 1),
        ("[SEP]", 2),
    ],
)

Common patterns:

# GPT-2: sentence <|endoftext|>
TemplateProcessing(
    single="$A <|endoftext|>",
    special_tokens=[("<|endoftext|>", 50256)]
)

# RoBERTa: <s> sentence </s>
TemplateProcessing(
    single="<s> $A </s>",
    pair="<s> $A </s> </s> $B </s>",
    special_tokens=[("<s>", 0), ("</s>", 2)]
)

Alignment tracking

Track token positions in original text:

output = tokenizer.encode("Hello, world!")

# Get token offsets
for token, offset in zip(output.tokens, output.offsets):
    start, end = offset
    print(f"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}")

# Output:
# hello      → [ 0,  5): 'Hello'
# ,          → [ 5,  6): ','
# world      → [ 7, 12): 'world'
# !          → [12, 13): '!'

Use cases:

Named entity recognition (map predictions back to text)
Question answering (extract answer spans)
Token classification (align labels to original positions)

Integration with transformers

Load with AutoTokenizer

from transformers import AutoTokenizer

# AutoTokenizer automatically uses fast tokenizers
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Check if using fast tokenizer
print(tokenizer.is_fast)  # True

# Access underlying tokenizers.Tokenizer
fast_tokenizer = tokenizer.backend_tokenizer
print(type(fast_tokenizer))  # <class 'tokenizers.Tokenizer'>

Convert custom tokenizer to transformers

from tokenizers import Tokenizer
from transformers import PreTrainedTokenizerFast

# Train custom tokenizer
tokenizer = Tokenizer(BPE())
# ... train tokenizer ...
tokenizer.save("my-tokenizer.json")

# Wrap for transformers
transformers_tokenizer = PreTrainedTokenizerFast(
    tokenizer_file="my-tokenizer.json",
    unk_token="[UNK]",
    pad_token="[PAD]",
    cls_token="[CLS]",
    sep_token="[SEP]",
    mask_token="[MASK]"
)

# Use like any transformers tokenizer
outputs = transformers_tokenizer(
    "Hello world",
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt"
)

Common patterns

Train from iterator (large datasets)

from datasets import load_dataset

# Load dataset
dataset = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")

# Create batch iterator
def batch_iterator(batch_size=1000):
    for i in range(0, len(dataset), batch_size):
        yield dataset[i:i + batch_size]["text"]

# Train tokenizer
tokenizer.train_from_iterator(
    batch_iterator(),
    trainer=trainer,
    length=len(dataset)  # For progress bar
)

Performance: Processes 1GB in ~10-20 minutes

Enable truncation and padding

# Enable truncation
tokenizer.enable_truncation(max_length=512)

# Enable padding
tokenizer.enable_padding(
    pad_id=tokenizer.token_to_id("[PAD]"),
    pad_token="[PAD]",
    length=512  # Fixed length, or None for batch max
)

# Encode with both
output = tokenizer.encode("This is a long sentence that will be truncated...")
print(len(output.ids))  # 512

Multi-processing

from tokenizers import Tokenizer
from multiprocessing import Pool

# Load tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

def encode_batch(texts):
    return tokenizer.encode_batch(texts)

# Process large corpus in parallel
with Pool(8) as pool:
    # Split corpus into chunks
    chunk_size = 1000
    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]

    # Encode in parallel
    results = pool.map(encode_batch, chunks)

Speedup: 5-8× with 8 cores

Performance benchmarks

Training speed

Corpus Size	BPE (30k vocab)	WordPiece (30k)	Unigram (8k)
10 MB	15 sec	18 sec	25 sec
100 MB	1.5 min	2 min	4 min
1 GB	15 min	20 min	40 min

Hardware: 16-core CPU, tested on English Wikipedia

Tokenization speed

Implementation	1 GB corpus	Throughput
Pure Python	~20 minutes	~50 MB/min
HF Tokenizers	~15 seconds	~4 GB/min
Speedup	80×	80×

Test: English text, average sentence length 20 words

Memory usage

Task	Memory
Load tokenizer	~10 MB
Train BPE (30k vocab)	~200 MB
Encode 1M sentences	~500 MB

Supported models

Pre-trained tokenizers available via from_pretrained():

BERT family:

bert-base-uncased, bert-large-cased
distilbert-base-uncased
roberta-base, roberta-large

GPT family:

gpt2, gpt2-medium, gpt2-large
distilgpt2

T5 family:

t5-small, t5-base, t5-large
google/flan-t5-xxl

Other:

facebook/bart-base, facebook/mbart-large-cc25
albert-base-v2, albert-xlarge-v2
xlm-roberta-base, xlm-roberta-large

Browse all: https://huggingface.co/models?library=tokenizers

References

Training Guide - Train custom tokenizers, configure trainers, handle large datasets
Algorithms Deep Dive - BPE, WordPiece, Unigram explained in detail
Pipeline Components - Normalizers, pre-tokenizers, post-processors, decoders
Transformers Integration - AutoTokenizer, PreTrainedTokenizerFast, special tokens

Resources

Docs: https://huggingface.co/docs/tokenizers
GitHub: https://github.com/huggingface/tokenizers ⭐ 9,000+
Version: 0.20.0+
Course: https://huggingface.co/learn/nlp-course/chapter6/1
Paper: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)

This skill ships inside the AI Research Skills plugin.

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add orchestra-research/ai-research-skills --skill "huggingface-tokenizers" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

What's inside

Files included with this skill.

View on GitHub

HuggingFace Tokenizers - Fast Tokenization for NLP

When to use HuggingFace Tokenizers

Quick start

Installation

Load pretrained tokenizer

Train custom BPE tokenizer

Batch encoding with padding

Tokenization algorithms

BPE (Byte-Pair Encoding)

WordPiece

Unigram

Tokenization pipeline

Normalization

Pre-tokenization

Post-processing

Alignment tracking

Integration with transformers

Load with AutoTokenizer

Convert custom tokenizer to transformers

Common patterns

Train from iterator (large datasets)

Enable truncation and padding

Multi-processing

Performance benchmarks

Training speed

Tokenization speed

Memory usage

Supported models

References

Resources

Local tool required

What's inside

More skills

academic-plotting

ara-compiler

ara-research-manager

ara-rigor-reviewer

audiocraft-audio-generation

autogpt-agents