How sarvamai/llm_intent_entity works
This package evaluates ASR transcriptions for preservation of intent and entities, rather than textual similarity. The main workflow loads a dataset, normalizes reference and predicted text, submits evaluation prompts to an LLM, computes aggregate metrics, and saves results and logs; Google Sheets upload is an optional output. The README also documents language support, caching, and failed-request reporting. Source excerpts establish the main orchestration, API request handling, and Sheets upload; links not directly shown in the excerpts are omitted.
Main entry points: main.py Evaluation orchestration · llm_api.py LLM request client · utilities.py Sheets upload
Components 9
Evaluation workflow
- Evaluation orchestration
src/llm_intent_entity/main.py
Language processing
- Text normalization
src/llm_intent_entity/utilities.py
LLM evaluation
- LLM request client
src/llm_intent_entity/llm_api.py
Results and reporting
- Intent and entity metrics
src/llm_intent_entity/utilities.py - CSV and evaluation logs
src/llm_intent_entity/main.py - Sheets upload
src/llm_intent_entity/utilities.py
Other components
- Dataset input
- Vertex AI
- Google Sheets
Connections 8
- Dataset input → Evaluation orchestration: provides dataset
README.md - Evaluation orchestration → Text normalization: normalizes text
src/llm_intent_entity/main.py - Evaluation orchestration → LLM request client: queues prompts
src/llm_intent_entity/main.py - LLM request client → Vertex AI: requests evaluations
- Evaluation orchestration → Intent and entity metrics: calculates scores
src/llm_intent_entity/main.py - Evaluation orchestration → CSV and evaluation logs: writes results
src/llm_intent_entity/main.py - Evaluation orchestration → Sheets upload: uploads results
src/llm_intent_entity/main.py - Sheets upload → Google Sheets: writes worksheets
src/llm_intent_entity/utilities.py