How huggingface/swift-transformers works
swift-transformers is a Swift library for using language models in apps, especially tokenization, Hugging Face Hub downloads, and CoreML inference and generation. A typical workflow loads tokenizer assets (from the Hub or local files), formats and tokenizes text or chat messages, runs a model to generate token IDs, then decodes them. The graph preserves the documented Hub, tokenizer, model, and generation capabilities. Some internal wiring is based on README and package-level evidence because the sampled source excerpts are partial.
Main entry points: Tokenizer.swift AutoTokenizer · LanguageModel.swift LanguageModel · Generation.swift Generation loop · PreTokenizer.swift Pre-tokenization
Components 16
Hub access
- Hub API
Sources/Hub/HubApi.swift - Hub downloads
Sources/Hub/Hub.swift - Hub config
Sources/Hub/Config.swift
Tokenization
- AutoTokenizer
Sources/Tokenizers/Tokenizer.swift - Pre-tokenization
Sources/Tokenizers/PreTokenizer.swift - Tokenizer algorithms
Sources/Tokenizers - Text processing
Sources/Tokenizers/Normalizer.swift - Chat templates
Sources/Tokenizers/Tokenizer.swift
Models
- LanguageModel
Sources/Models/LanguageModel.swift - Weight loading
Sources/Models/Weights.swift
Generation
- Generation loop
Sources/Generation/Generation.swift - Logits processors
Sources/Generation/LogitsWarper/LogitsProcessor.swift - Token selection
Sources/Generation/Decoders.swift
Other components
- Swift app
- Hugging Face Hub
- CoreML runtime
Connections 19
- Swift app → AutoTokenizer: loads tokenizer
- AutoTokenizer → Hub config: requests assets
- Hub config → Hub downloads: resolves files
- Hub downloads → Hub API: downloads files
- Hub API → Hugging Face Hub: requests files
- AutoTokenizer → Pre-tokenization: splits text
- Pre-tokenization → Text processing: normalizes text
- Pre-tokenization → Tokenizer algorithms: feeds segments
- Swift app → Chat templates: formats messages
- Chat templates → AutoTokenizer: encodes template
- Swift app → LanguageModel: loads model
- Swift app → Weight loading: loads weights
- LanguageModel → CoreML runtime: runs inference
- Swift app → LanguageModel: requests generation
- LanguageModel → Generation loop: delegates generation
- Generation loop → LanguageModel: requests logits
- Generation loop → Logits processors: processes logits
- Generation loop → Token selection: selects token
- Swift app → AutoTokenizer: decodes output