# arnnv/shl-recommendation-system architecture

> Architecture diagram of the GitHub repository arnnv/shl-recommendation-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arnnv/shl-recommendation-system
- Repository: https://github.com/arnnv/shl-recommendation-system

## Overview

##### Project type and purpose
- This is a full-stack, microservice-style recommendation system (“SHL Assessment Recommender”) that uses a FastAPI backend to serve recommendations and a Streamlit frontend for user interaction.
- It includes an auxiliary crawler service to ingest SHL assessment data, a hybrid search engine (vector + BM25), and an LLM-powered orchestration layer (LangGraph + LangChain).

##### File structure
- Top-level:
- app.py (FastAPI entry point)
- main.py (core LangGraph workflow / recommendation engine)
- crawler/ (crawler.py plus JSON data and state files)
- frontend/ (Streamlit app, requirements, Dockerfile)
- Dockerfile, .env.example, requirements.txt (backend Docker & dependencies)
- Patterns:
- Microservices: separate backend and frontend directories with their own Dockerfiles
- Clear separation of concerns: crawler service, recommendation logic, API layer, UI layer
- Deployment: Dockerfile and Procfile indicate containerized deployment; .env for secrets

##### README for additional insights
- Hybrid retrieval combining FAISS-via-Qdrant and BM25
- LangGraph orchestrates multi-step flows: parse job description → retrieve → rerank → respond
- External dependencies: Qdrant vector DB, OpenAI Embeddings, Anthropic / Claude LLMs
- Streaming: Uvicorn as ASGI server for FastAPI, Streamlit for UI

##### Detail
- Every major file/service folder as a component (crawler, frontend, backend, data stores)
- Docker and env var configuration as deployment details if needed
- Don’t over-abstract; show each moving part and its interactions to reflect the real architecture accurately

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User (Web Browser)"]:::user

    %% Frontend Layer
    subgraph "Frontend Service" 
        direction TB
        Streamlit["Streamlit App"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Service"
        direction TB
        API["FastAPI Service"]:::api
        Orchestrator["Recommendation Engine / LangGraph Orchestrator"]:::orchestrator
    end

    %% Hybrid Search Layer
    subgraph "Hybrid Search Layer"
        direction TB
        Qdrant["Qdrant Vector DB"]:::db
        BM25["BM25 Text Index"]:::db
        LocalJSON["Local JSON Store"]:::db
    end

    %% External LLM Services
    subgraph "External LLM Providers"
        direction TB
        OpenAI["OpenAI Embeddings & Models"]:::external
        Anthropic["Anthropic / Claude LLMs"]:::external
    end

    %% Crawler Service
    subgraph "Crawler Microservice"
        direction TB
        CrawlerApp["Crawler Script"]:::crawler
        JSONFull["shl_assessments.json"]:::data
        JSONPartial["shl_assessments_partial.json"]:::data
        CrawlState["shl_crawl_state.json"]:::data
    end

    %% Deployment & Config
    subgraph "Deployment Configuration"
        direction TB
        ReqBackend["requirements.txt"]:::config
        DockerBackend["Dockerfile"]:::config
        Procfile["Procfile"]:::config
        EnvFile[".env.example"]:::config
        Runtime["runtime.txt"]:::config
        README["README.md"]:::docs
        Approach["approach.md"]:::docs
    end

    %% Connections
    User -->|"inputs job description"| Streamlit
    Streamlit -->|"POST /recommend"| API
    API -->|"invoke"| Orchestrator
    Orchestrator -->|"vector search"| Qdrant
    Orchestrator -->|"sparse search"| BM25
    Orchestrator -->|"load data"| LocalJSON
    Orchestrator -->|"call embeddings & parsing"| OpenAI
    Orchestrator -->|"call reranker"| Anthropic
    Orchestrator -->|"return recommendations"| API
    API -->|"response"| Streamlit
    Streamlit -->|"display results"| User

    CrawlerApp -->|"writes full data"| JSONFull
    CrawlerApp -->|"writes partial data"| JSONPartial
    CrawlerApp -->|"updates state"| CrawlState
    CrawlerApp -->|"ingest to vector DB"| Qdrant

    %% Click Events
    click API "https://github.com/arnnv/shl-recommendation-system/blob/main/app.py"
    click Orchestrator "https://github.com/arnnv/shl-recommendation-system/blob/main/main.py"
    click CrawlerApp "https://github.com/arnnv/shl-recommendation-system/blob/main/crawler/crawler.py"
    click JSONFull "https://github.com/arnnv/shl-recommendation-system/blob/main/crawler/shl_assessments.json"
    click JSONPartial "https://github.com/arnnv/shl-recommendation-system/blob/main/crawler/shl_assessments_partial.json"
    click CrawlState "https://github.com/arnnv/shl-recommendation-system/blob/main/crawler/shl_crawl_state.json"
    click Streamlit "https://github.com/arnnv/shl-recommendation-system/blob/main/frontend/streamlit_app.py"
    click DockerBackend "https://github.com/arnnv/shl-recommendation-system/tree/main/frontend/Dockerfile"
    click ReqFrontend "https://github.com/arnnv/shl-recommendation-system/blob/main/frontend/requirements.txt"
    click LocalJSON "https://github.com/arnnv/shl-recommendation-system/blob/main/shl_assessments.json"
    click ReqBackend "https://github.com/arnnv/shl-recommendation-system/blob/main/requirements.txt"
    click DockerBackend "https://github.com/arnnv/shl-recommendation-system/tree/main/Dockerfile"
    click Procfile "https://github.com/arnnv/shl-recommendation-system/tree/main/Procfile"
    click EnvFile "https://github.com/arnnv/shl-recommendation-system/blob/main/.env.example"
    click Runtime "https://github.com/arnnv/shl-recommendation-system/blob/main/runtime.txt"
    click README "https://github.com/arnnv/shl-recommendation-system/blob/main/README.md"
    click Approach "https://github.com/arnnv/shl-recommendation-system/blob/main/approach.md"

    %% Styles
    classDef user fill:#D0E1FF,stroke:#4D7EA8,stroke-width:2px
    classDef frontend fill:#E0F7FA,stroke:#0097A7,stroke-width:2px
    classDef api fill:#E8F5E9,stroke:#388E3C,stroke-width:2px
    classDef orchestrator fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px
    classDef db fill:#FFF3E0,stroke:#FB8C00,stroke-width:2px
    classDef external fill:#F3E5F5,stroke:#8E24AA,stroke-width:2px
    classDef crawler fill:#FFF9C4,stroke:#F9A825,stroke-width:2px
    classDef data fill:#FFEBEE,stroke:#E53935,stroke-width:2px
    classDef config fill:#ECEFF1,stroke:#546E7A,stroke-width:2px
    classDef docs fill:#F1F8E9,stroke:#7CB342,stroke-width:2px
```
