Home / Skills / sentencepiece

sentencepiece

Orchestra-Research/AI-Research-SKILLsSkill

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

Scanned · 1 to knowSkillSource: GitHub
13,721 stars Updated 25d ago License MIT Context ≈88 tokens always loaded
77popular

Why this score

Stars22/35
Usage (forks)9/15
Upkeep25/25
License15/15
Owner history6/10

This repo hosts 10 skills, so its 13,721 stars are split between them (about 1,372 each) for scoring.

Popularity score: 77/100 from public GitHub signals only. It shows how widely used and maintained this is, not how well it works. Not included yet: the fake-star check. The security scan is shown separately and doesn't change the score. How we rank

Security

Scanned Oct 20, 2026 at commit 773a52944ba4, with Cisco's open-source skill-scanner.

What its code does:

  • Uses dependencies without a pinned version A later version of a dependency could change what runs.SKILL.md and 1 more

A scan lists patterns in the code. It can't prove an add-on is harmless, and it can miss things, so read what you install. How we scan

Alternatives

Nothing from another maker does quite the same job yet.