How sh4shv4t/inference-forge works
Inference Forge is an async support-ticket classification service. Clients submit ticket batches to the FastAPI API, which validates input, deduplicates tickets, estimates complexity and ETA, then orchestrates priority-aware adaptive batching and Sarvam inference. Results and job progress are stored in Redis and exposed through polling and SSE; partial failures are isolated. Supporting layers provide circuit breaking, retries, caching, configuration, logging, and metrics. The README documents the overall processing stages; some route-to-pipeline details are not visible in the excerpts, so those documented links cite the README.
Main entry points: routes.py API routes · caller.py Sarvam caller · main.py Application setup · cache.py Ticket deduplication
Components 15
API and responses
- Application setup
src/inference_forge/main.py - API routes
src/inference_forge/api/routes.py - Request schemas
src/inference_forge/api/schemas.py
Inference pipeline
- Ticket deduplication
src/inference_forge/pipeline/cache.py - Complexity and ETA
src/inference_forge/pipeline/estimator.py - Priority batcher
src/inference_forge/pipeline/batcher.py - Sarvam caller
src/inference_forge/pipeline/caller.py - Circuit breaker
src/inference_forge/pipeline/circuit_breaker.py
State and persistence
- Job state and events
src/inference_forge/pipeline/job_store.py - Redis
Configuration and observability
- Runtime settings
src/inference_forge/config.py - Structured logging
src/inference_forge/observability/logger.py - Metrics and health
src/inference_forge/observability/metrics.py
Other components
- API client
- Sarvam AI API
Connections 25
- API client → API routes: submits and queries
README.md - Application setup → API routes: registers routes
src/inference_forge/main.py - Application setup → Runtime settings: reads settings
src/inference_forge/main.py - Application setup → Ticket deduplication: creates cache
src/inference_forge/main.py - Application setup → Job state and events: creates store
src/inference_forge/main.py - Application setup → Sarvam caller: creates caller
src/inference_forge/main.py - API routes → Request schemas: uses models
src/inference_forge/api/routes.py - API routes → Ticket deduplication: checks tickets
src/inference_forge/api/routes.py - API routes → Complexity and ETA: classifies and estimates
README.md - API routes → Priority batcher: orchestrates batches
README.md - API routes → Sarvam caller: processes batches
README.md - API routes → Job state and events: updates and reads jobs
src/inference_forge/api/routes.py - API routes → Metrics and health: serves metrics
src/inference_forge/api/routes.py - Priority batcher → Complexity and ETA: classifies tickets
src/inference_forge/pipeline/batcher.py - Sarvam caller → Ticket deduplication: reads and caches results
src/inference_forge/pipeline/caller.py - Sarvam caller → Circuit breaker: guards API calls
src/inference_forge/pipeline/caller.py - Sarvam caller → Sarvam AI API: sends inference requests
src/inference_forge/pipeline/caller.py - Ticket deduplication → Redis: reads and writes cache
src/inference_forge/pipeline/cache.py - Job state and events → Redis: stores jobs and publishes events
src/inference_forge/pipeline/job_store.py - API routes → Job state and events: streams job events
README.md - Sarvam caller → Metrics and health: records API metrics
src/inference_forge/pipeline/caller.py - Ticket deduplication → Metrics and health: records cache metrics
src/inference_forge/pipeline/cache.py - Circuit breaker → Metrics and health: records breaker state
src/inference_forge/pipeline/circuit_breaker.py - API routes → Structured logging: logs job activity
src/inference_forge/api/routes.py - Request schemas → Runtime settings: uses validation limits
src/inference_forge/api/schemas.py