How prashantcsuthar-cmd/inferencedevs works
ICE constrains local language-model generation by tracking a JSON grammar state, finding vocabulary tokens admissible in that state, masking other logits, and advancing the state after each generated token. The graph separates schema/grammar, token compatibility, decoding, and model loading. The README also describes diagnostics, validation, and a dashboard, but those paths are absent from the supplied tree, so they are not represented. The shown main.py excerpt is empty; runtime connections are limited to those visible in source.
Main entry points: engine.py Incremental decoder · compatibility.py Admissibility engine · grammar.py JSON transitions · states.py Grammar state
Components 10
Schema and grammar
- Schema compiler
src/schema/parser.py - JSON transitions
src/schema/grammar.py - Grammar state
src/schema/states.py
Token compatibility
- Admissibility engine
src/tokenizer/compatibility.py - Token prefix trie
src/tokenizer/trie.py - Token vocabulary
src/tokenizer/vocabulary.py
Constrained decoding
- Incremental decoder
src/decoding/engine.py - Logit masker
src/decoding/logits_processor.py
Model loading
- Qwen loader
src/models/loader.py
Other components
- Caller
Connections 13
- Caller → Incremental decoder: requests generation
- Incremental decoder → Logit masker: applies mask
src/decoding/engine.py - Incremental decoder → Admissibility engine: gets valid tokens
src/decoding/engine.py - Incremental decoder → JSON transitions: advances state
src/decoding/engine.py - Incremental decoder → Grammar state: starts state
src/decoding/engine.py - Incremental decoder → Qwen loader: uses loaded model
- Logit masker → Admissibility engine: gets valid tokens
src/decoding/logits_processor.py - Admissibility engine → JSON transitions: tests transitions
src/tokenizer/compatibility.py - Admissibility engine → Token prefix trie: builds and traverses
src/tokenizer/compatibility.py - Admissibility engine → Token vocabulary: builds and reads
src/tokenizer/compatibility.py - Schema compiler → Grammar state: creates initial state
src/schema/parser.py - JSON transitions → Grammar state: uses state types
src/schema/grammar.py - Token prefix trie → Token vocabulary: indexes vocabulary
src/tokenizer/trie.py