huggingface-tokenizers

Fast tokenizers optimized for research and production.

What this skill does for you

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Quoted from the skill description in the pinned source

Who made it

Publisherdavila7

Repositoryclaude-code-templates

Installs349as of Sep 27, 2026

Use it with Edge

Set up Edge for me: read getedge.cc/SKILL.md and follow it. Then use Edge to load the huggingface-tokenizers skill from davila7/claude-code-templates.

Paste it into Claude, ChatGPT, Codex or Cursor.

Or read the skill source first.

Security evidence

Security state at 2026-09-30: reviewed. State definitions. This records static evidence for this revision; live load eligibility is checked again.

3 static scans, 0 counted findings. The listed scanners recorded the results shown for this package revision.

Scanner	Date	Result
Edge static checks	2026-09-27	Pass
Cisco skill-scanner	2026-09-27	Pass
Semgrep Edge rules	2026-09-27	Pass

Static scans check the code, not how well the skill works.

Scanner scope, raw findings and mirrored provider records

Edge static checks (Edge-run): edge-static/1.0.1. Scope: Static patterns for remote execution, credentials with network sends, obfuscation, prompt overrides, hidden Unicode, binaries and persistence. Revision: 6af7cdf57b810aca5e544a0f695bf058f1c2a744. Raw findings: . Counted findings: .

Cisco skill-scanner (Edge-run): cisco-skill-scanner/2.1.0. Scope: Local static, YARA, pipeline and behavioral analyzers; no LLM or AI Defense analysis. Revision: 6af7cdf57b810aca5e544a0f695bf058f1c2a744. Raw findings: 3. Counted findings: .

Semgrep Edge rules (Edge-run): semgrep/1.178.0+edge-rules.c15016a36fd3+offline-v1. Scope: Six Edge-authored static rules with Semgrep CE; offline execution and metrics disabled. Semgrep-maintained rules excluded. Revision: 6af7cdf57b810aca5e544a0f695bf058f1c2a744. Raw findings: . Counted findings: .

Source and licence

Repository: https://github.com/davila7/claude-code-templates
Pinned skill file: View source at revision
Revision: 6af7cdf57b810aca5e544a0f695bf058f1c2a744
Package hash: sha256:893c0c8c0fa07650a33a0f0f45eca57a738edf769aaf217c7aa4568168f628e1
Licence: MIT