How rishabhjain2003/shl-copilot works
1. System purpose
- This repo is a stateless conversational recommender for SHL Individual Test Solutions.
- It takes a chat history, classifies the user’s intent, retrieves candidate assessments from a local catalog, and returns grounded recommendations.
2. Runtime and service boundary
- The online service is a FastAPI app in app/main.py.
- Exposed endpoints are minimal: a lightweight health check and a chat endpoint.
- The app is designed to run in a container via Dockerfile and Procfile, with deployment settings for platforms like Fly.io.
3. Conversation orchestration
- The core control plane is an explicit state machine implemented across app/state.py and prompt/model definitions.
- The readme names five branches: CLARIFY, RETRIEVE_AND_RECOMMEND, REFINE, COMPARE, and REFUSE.
- This means the LLM is not free-form end-to-end; it is steered into one of a small number of action paths.
4. Request-to-response flow
- A /chat request carries the full message list.
- The app interprets the latest user intent, decides the action branch, optionally retrieves catalog candidates, then generates a response and structured recommendation list.
- The response includes both natural-language text and machine-readable recommendation objects, plus an end_of_conversation flag.
5. Retrieval subsystem
- app/retrieval.py is the main matching layer.
- It uses hybrid retrieval: TF-IDF lexical search plus sentence-transformer semantic embeddings, combined with a weighted fusion.
- The repository includes prebuilt indices in data/tfidf.pkl and data/embeddings.npy, so retrieval runs locally without needing external search infrastructure.
6. Catalog and grounding
- app/catalog.py and data/catalog.json represent the SHL product catalog as the authoritative reference set.
- The system enforces grounding by validating URLs and restricting final selection to retrieved catalog candidates.
- The LLM is therefore a selector and summarizer, not the source of truth for product identity.
7. LLM integration and failover
- app/llm.py handles model invocation and provider fallback.
- The readme describes a dynamic query-time failover chain: Groq first, then Gemini Flash, then OpenAI GPT-4o-mini on rate-limit or availability failure.
- This creates a latency/availability boundary between the app and external model providers.
8. Prompting and response shape
- app/prompts.py contains separate prompts per action, matching the state-machine branches.
- app/models.py likely defines request/response schemas and typed internal objects used by FastAPI and the orchestration layer.
- This separation keeps intent routing, retrieval, and answer generation decoupled.
9. Security and validation boundary
- app/security.py is the likely place for URL, catalog, or output validation rules.
- The architecture emphasizes programmatic grounding checks after generation, rather than trusting the model output directly.
- That boundary is important: external LLM output is filtered before being returned to the client.
10. Offline build pipeline
- scripts/scrape.py populates the catalog from SHL sources into the local dataset.
- scripts/build_embeddings.py builds the semantic and lexical artifacts used at runtime.
- This means the service is mostly read-only at runtime; expensive preparation happens offline.
11. Packaging and startup behavior
- The container is optimized to avoid startup downloads by baking model caches into the image.
- README.md notes HF_HOME=/app/.cache, and startup eagerly loads embedding resources so the first request does not pay model-loading latency.
- This affects the deployment architecture more than the application code does.
12. Testing and evaluation
- tests/test_api.py covers endpoint behavior, tests/test_behavior.py probes decision behavior, and tests/test_traces.py replays traces with Recall@10.
- The presence of trace replay and recall measurement shows the system is evaluated as a retrieval-and-recommendation pipeline, not just a chat app.
13. Data and sample assets
- sample_conversations/ contains example dialogue traces used to validate or demonstrate the state machine and retrieval behavior.
- data/ contains the runtime artifacts that the service depends on directly, making the repo self-contained after preprocessing.
14. High-level architecture summary
- The overall structure is: FastAPI request layer → state-machine intent routing → hybrid retrieval over local catalog/indexes → LLM generation with provider failover → grounding/validation → structured chat response.
- The key architectural constraint is that all recommendation decisions remain tied to the local catalog and retrieval results, with external LLMs used only as controlled reasoning/generation components.
Main entry points: state.py State machine · retrieval.py Hybrid retrieval · main.py FastAPI app · catalog.py Catalog layer
Components 17
Runtime
Online API, orchestration, retrieval, and validation
- FastAPI app · Entry point with chat and health endpoints
app/main.py - State machine · Routes each chat turn into one of the action branches
app/state.py - Branch prompts · Prompt templates for CLARIFY, RETRIEVE_AND_RECOMMEND, REFINE, COMPARE, and REFUSE
app/prompts.py - Schemas · Request, response, and internal data structures
app/models.py - Hybrid retrieval · Combines lexical and semantic candidate ranking
app/retrieval.py - Catalog layer · Loads and serves the authoritative product catalog
app/catalog.py - LLM gateway · Invokes models with Groq to Gemini to OpenAI fallback
app/llm.py - Validation · Validates URLs and filters model output against catalog rules
app/security.py
Data
Local catalog and prebuilt retrieval artifacts
- Catalog JSON · Local authoritative assessment catalog
data/catalog.json - TF-IDF index · Prebuilt lexical retrieval index
data/tfidf.pkl - Embeddings · Prebuilt sentence-transformer vectors
data/embeddings.npy
Offline prep
Scraping and embedding build pipeline
- Scraper · Builds the local catalog from SHL sources
scripts/scrape.py - Embedding build · Generates retrieval artifacts for runtime
scripts/build_embeddings.py
Evaluation
Behavioral and trace-based checks
- API tests · Exercises service behavior and responses
tests/test_api.py - Behavior tests · Probes routing and recommendation behavior
tests/test_behavior.py - Trace replay · Replays conversations and measures Recall@10
tests/test_traces.py
Examples
Sample conversation traces
- Sample traces · Example dialogues used to inspect the system
sample_conversations/GenAI_SampleConversations
Connections 20
- FastAPI app → Schemas: uses schemas · FastAPI request and response typing
- FastAPI app → State machine: routes chat · Turns incoming messages into a controlled action branch
- State machine → Branch prompts: selects prompt · Chooses the branch-specific prompt template
- State machine → Hybrid retrieval: needs candidates · Retrieves catalog matches when the branch requires grounding
- State machine → LLM gateway: generates reply · Produces the final natural-language response
- Hybrid retrieval → Catalog layer: reads catalog · Retrieval operates over the local product set
- Catalog layer → Catalog JSON: loads data · Authoritative catalog source
- Hybrid retrieval → TF-IDF index: lexical search · TF-IDF candidate scoring
- Hybrid retrieval → Embeddings: semantic search · Embedding-based similarity search
- LLM gateway → Validation: validated by · Model output is checked before returning
- Validation → Catalog layer: checks grounding · Constrains recommendations to catalog-backed items
- FastAPI app → Validation: guards response · Final API response passes validation and grounding checks
- Scraper → Catalog JSON: builds catalog · Populates the local dataset from external sources
- Embedding build → TF-IDF index: builds index · Creates lexical retrieval artifacts
- Embedding build → Embeddings: builds vectors · Creates semantic retrieval artifacts
- Sample traces → Behavior tests: feeds examples · Example dialogue traces support behavioral inspection
- Sample traces → Trace replay: replays traces · Used for evaluation against known conversations
- API tests → FastAPI app: tests endpoints · Verifies service contract
- Behavior tests → State machine: tests routing · Checks branch selection and state transitions
- Trace replay → Hybrid retrieval: measures recall · Evaluates retrieval quality over recorded traces