Home / Skills / huggingface-tokenizers
huggingface-tokenizers
Orchestra-Research/AI-Research-SKILLsSkill
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
Why this score
This repo hosts 10 skills, so its 13,721 stars are split between them (about 1,372 each) for scoring.
Popularity score: 77/100 from public GitHub signals only. It shows how widely used and maintained this is, not how well it works. Not included yet: the fake-star check. The security scan is shown separately and doesn't change the score. How we rank
Security
Scanned Oct 20, 2026 at commit 773a52944ba4, with Cisco's open-source skill-scanner.
What its code does:
- Uses dependencies without a pinned version A later version of a dependency could change what runs.SKILL.md and 2 more
A scan lists patterns in the code. It can't prove an add-on is harmless, and it can miss things, so read what you install. How we scan
Alternatives
Other makers' add-ons that do the same job, next to this one.
| Name | Score | Stars | Context | Updated |
|---|---|---|---|---|
| This onehuggingface-tokenizers Orchestra-Research/AI-Research-SKILLs · Skill | 77 | ★ 14k | ≈93 always | 25d ago |
| huggingface-best huggingface/skills · Skill Both about huggingface | 79 | ★ 11k | ≈25 always | 27d ago |
| huggingface-accelerate OpenRaiser/NanoResearch · Skill Both about huggingface and training | 70 | ★ 1.3k | ≈74 always | 26d ago |