How microsoft/llmlingua works
LLMLingua is a Python library that compresses prompts before they are sent to an LLM. Its principal workflow accepts prompt context and task text, uses a compact model and token-level analysis to select or preserve important content, and returns a shorter prompt. The README also documents a LongLLMLingua mode for long-context and retrieval scenarios and a faster LLMLingua-2 mode. The sampled implementation excerpt is incomplete, so internal method-level wiring is shown only where supported; experiment and training scripts are not part of the runtime map.
Main entry points: prompt_compressor.py PromptCompressor · utils.py Token utilities
Components 11
Prompt compression
- PromptCompressor
llmlingua/prompt_compressor.py - Token analysis
llmlingua/prompt_compressor.py - Context ranking
llmlingua/prompt_compressor.py - LongLLMLingua mode
llmlingua/prompt_compressor.py - LLMLingua-2 mode
llmlingua/prompt_compressor.py
Model support
- Token utilities
llmlingua/utils.py - Compact model
- Tokenizer
- Embedding API
Integration
- Downstream LLM
Other components
- Application
Connections 11
- Application → PromptCompressor: submits prompt
- PromptCompressor → Tokenizer: tokenizes text
- PromptCompressor → Compact model: loads scores
- PromptCompressor → Token analysis: analyzes tokens
- Token analysis → Token utilities: uses utilities
- PromptCompressor → Context ranking: ranks context
- PromptCompressor → LongLLMLingua mode: selects mode
- PromptCompressor → LLMLingua-2 mode: selects mode
- PromptCompressor → Embedding API: requests embeddings
- PromptCompressor → Application: returns compressed prompt
- Application → Downstream LLM: sends prompt