# 2003harsh/skillhorizon architecture

> Architecture diagram of the GitHub repository 2003harsh/skillhorizon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/2003harsh/skillhorizon
- Repository: https://github.com/2003harsh/skillhorizon

## Overview

##### Project Type and Purpose
- From the file tree and README, this is a Python-based backend pipeline (CLI/service) for an AI-powered course recommendation system.
- It orchestrates:
- Job listing scraping (Playwright, Scrapy)
- Skill-gap analysis
- Course data scraping
- Vector embedding & semantic search (FAISS + LangChain)
- LLM reasoning for personalized recommendations
- Data storage in MongoDB

##### File Structure
- Top-level:
- main.py – entry point, coordinates the pipeline
- requirements.txt – dependencies (LangChain, FAISS, PyMongo, Scrapy, Playwright, etc.)
- utils/ – modular scripts for each pipeline stage:
– get_jobs_json.py: scrape & extract required skills from jobs
– formatted_json_generator.py: normalize/sanitize scraped data
– convertor.py: convert JSON to embeddings input
– create_index.py: build/manage FAISS vector index
– genai.py: orchestrate LLM calls for ranking & explanation
– reterive_output.py: aggregate results and format output

##### README for Additional Insights
- Workflow stages clearly defined: Job & skill-gap analysis → Course scraping & ranking → Semantic search + LLM personalization.
- External systems: Coursera, Udemy, edX (scraping targets); LLM provider; MongoDB.

a. Main Components
1. User Interface (CLI or API endpoint in main.py)
2. Job Scraper Service (Playwright & Scrapy)
3. Skill-Gap Analyzer (compares scraped skills vs. user skills)
4. Course Scraper Service (Scrapy for course & review data)
5. Data Transformation Layer (formatted_json_generator + convertor)
6. Vector Store (FAISS index managed by create_index)
7. Retrieval & LLM Pipeline (LangChain + genai)
8. Database (MongoDB for raw & processed data)
9. External Services:
- Job boards (LinkedIn, Indeed, etc.)
- Course platforms (Coursera, Udemy, edX)
- LLM API (OpenAI, Cohere, etc.)

b. Relationships & Interactions
- User → main.py: submits profile & preferences
- main.py → Job Scraper: fetch listings → MongoDB
- main.py → Skill-Gap Analyzer: reads from MongoDB → outputs missing skills
- main.py → Course Scraper: fetch course data & reviews → MongoDB
- Data Transformation Layer: reads raw data → outputs normalized JSON
- Vector Store Indexing: ingest normalized JSON → builds/updates FAISS index
- Retrieval Pipeline:
- FAISS search for relevant courses
- genai calls LLM to refine & rank results
- main.py → retrieve_output: collects final recommendations → user

c. Architectural Patterns & Design Principles
- Pipeline / staged ETL pattern
- Modular single-responsibility scripts (utils/)
- Separation of concerns: scraping vs. transformation vs. retrieval
- Vector-based RAG architecture for semantic search + LLM

d. Key Technologies & Frameworks
- Python modules in utils/
- Scrapy & Playwright for ingestion
- MongoDB for persistence
- FAISS + LangChain for vector search & RAG
- LLM provider for natural language reasoning

##### Details to Capture
- Entry point (main.py) orchestrates everything.
- Each utils script’s responsibility.
- Data persisted in MongoDB vs. indexed in FAISS vs. runtime LLM calls.
- External web scraping targets and LLM endpoint.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        User["User (CLI)"]:::ingestion
        Main["main.py"]:::ingestion
        JobScraper["Job Scraper\n(Playwright & Scrapy)"]:::ingestion
        CourseScraper["Course Scraper\n(Scrapy)"]:::ingestion
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        SkillGap["Skill-Gap Analyzer"]:::processing
        Normalize["Data Normalization\n(utils/formatted_json_generator)"]:::processing
        Convertor["JSON ➔ Embeddings Converter\n(utils/convertor)"]:::processing
        IndexMgr["FAISS Vector Index Manager\n(utils/create_index)"]:::processing
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        MongoDB[(MongoDB)]:::storage
        VectorStore[(FAISS Vector Store)]:::storage
    end

    %% Retrieval Layer
    subgraph "Retrieval Layer"
        direction TB
        GenAI["Retrieval & LLM Pipeline\n(utils/genai)"]:::retrieval
        OutputAggr["Output Aggregator\n(utils/reterive_output)"]:::retrieval
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        JobBoards["Job Boards\n(LinkedIn, Indeed)"]:::external
        CoursePlatforms["Course Platforms\n(Coursera, Udemy, edX)"]:::external
        LLMAPI["LLM API\n(OpenAI, Cohere)"]:::external
    end

    %% Data Flow
    User -->|"run pipeline"| Main
    Main -->|"scrape jobs"| JobScraper
    JobScraper -->|"raw jobs JSON"| MongoDB
    Main -->|"analyze skills"| SkillGap
    SkillGap -->|"missing skills JSON"| MongoDB
    Main -->|"scrape courses"| CourseScraper
    CourseScraper -->|"raw courses JSON"| MongoDB

    MongoDB -->|"fetch raw data"| Normalize
    Normalize -->|"normalized JSON"| Convertor
    Convertor -->|"embeddings input"| IndexMgr
    IndexMgr -->|"update index"| VectorStore

    VectorStore -->|"semantic search"| GenAI
    LLMAPI -->|"LLM calls"| GenAI
    GenAI -->|"ranked results"| OutputAggr
    OutputAggr -->|"final recommendations"| User

    %% External connections
    JobScraper --> JobBoards
    CourseScraper --> CoursePlatforms

    %% Click Events
    click Main "https://github.com/2003harsh/skillhorizon/blob/main/main.py"
    click JobScraper "https://github.com/2003harsh/skillhorizon/blob/main/utils/get_jobs_json.py"
    click Normalize "https://github.com/2003harsh/skillhorizon/blob/main/utils/formatted_json_generator.py"
    click Convertor "https://github.com/2003harsh/skillhorizon/blob/main/utils/convertor.py"
    click IndexMgr "https://github.com/2003harsh/skillhorizon/blob/main/utils/create_index.py"
    click GenAI "https://github.com/2003harsh/skillhorizon/blob/main/utils/genai.py"
    click OutputAggr "https://github.com/2003harsh/skillhorizon/blob/main/utils/reterive_output.py"

    %% Styles
    classDef ingestion fill:#D6EAF8,stroke:#1F618D,color:#1F618D;
    classDef processing fill:#D5F5E3,stroke:#196F3D,color:#196F3D;
    classDef storage fill:#FCF3CF,stroke:#7D6608,color:#7D6608;
    classDef retrieval fill:#FADBD8,stroke:#C0392B,color:#C0392B;
    classDef external fill:#E8DAEF,stroke:#6C3483,color:#6C3483;
```
