sentencepiece

by firecrawl

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory),… npx skills add https://github.com/firecrawl/ai-research-skills --skill sentencepiece Download ZIPGitHub11

SentencePiece - Language-Independent Tokenization

Unsupervised tokenizer that works on raw text without language-specific preprocessing.

When to use SentencePiece

Use SentencePiece when:

• Building multilingual models (no language-specific rules)

• Working with CJK languages (Chinese, Japanese, Korean)

• Need reproducible tokenization (deterministic vocabulary)

• Want to train on raw text (no pre-tokenization needed)

• Require lightweight deployment (6MB memory, 50k sentences/sec)

Performance:

• Speed: 50,000 sentences/sec

• Memory: ~6MB for loaded model

• Languages: All (language-independent)

Use alternatives instead:

• HuggingFace Tokenizers: Faster training, more flexibility

• tiktoken: OpenAI models (GPT-3.5/4)

• BERT WordPiece: English-centric tasks

Language-independent design

Whitespace as symbol (▁)

text = "Hello world"
pieces = sp.encode(text, out_type=str)
print(pieces) # ['▁Hello', '▁world']

# Decode preserves spaces
decoded = sp.decode_pieces(pieces)
print(decoded) # "Hello world"

Key principle: Treat text as raw Unicode, whitespace = ▁ (meta symbol)

Tokenization algorithms

BPE (Byte-Pair Encoding)

spm.SentencePieceTrainer.train(
 input='data.txt',
 model_prefix='bpe_model',
 vocab_size=16000,
 model_type='bpe'
)

Used by: mBART

Unigram (default)

spm.SentencePieceTrainer.train(
 input='data.txt',
 model_prefix='unigram_model',
 vocab_size=8000,
 model_type='unigram'
)

Used by: T5, ALBERT, XLNet

Encoding options

Subword regularization

# Sample different tokenizations
for _ in range(3):
 pieces = sp.encode('tokenization', out_type=str, enable_sampling=True, alpha=0.1)
 print(pieces)

# Output (different each time):
# ['▁token', 'ization']
# ['▁tok', 'en', 'ization']

Use case: Data augmentation for robustness.

Common patterns

T5-style training

spm.SentencePieceTrainer.train(
 input='c4_corpus.txt',
 model_prefix='t5',
 vocab_size=32000,
 model_type='unigram',
 user_defined_symbols=[f' ' for i in range(100)],
 unk_id=2,
 eos_id=1,
 pad_id=0
)

Integration with transformers

from transformers import T5Tokenizer

# T5 uses SentencePiece internally
tokenizer = T5Tokenizer.from_pretrained('t5-base')
inputs = tokenizer('translate English to French: Hello', return_tensors='pt')

Performance benchmarks

Training speed

Corpus BPE (16k) Unigram (8k) 100 MB 1-2 min 3-4 min 1 GB 10-15 min 30-40 min

Tokenization speed

• SentencePiece: 50,000 sentences/sec

• HF Tokenizers: 200,000 sentences/sec (4× faster)

Supported models

T5 family: t5-base, t5-large (32k vocab, Unigram) ALBERT: albert-base-v2 (30k vocab, Unigram) XLNet: xlnet-base-cased (32k vocab, Unigram) mBART: facebook/mbart-large-50 (250k vocab, BPE)

References

• Training Guide - Detailed options, corpus preparation

• Algorithms - BPE vs Unigram, subword regularization

Resources

• GitHub: https://github.com/google/sentencepiece ⭐ 10,000+

• Paper: https://arxiv.org/abs/1808.06226 (EMNLP 2018)

• Version: 0.2.0+