# stellarphantasm30/hack_the_future_hackathon architecture

> Architecture diagram of the GitHub repository stellarphantasm30/hack_the_future_hackathon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stellarphantasm30/hack_the_future_hackathon
- Repository: https://github.com/stellarphantasm30/hack_the_future_hackathon

## Overview

##### Project Type & Purpose
- This is a _data‑processing & AI‑driven pipeline_ with a separate visualization frontend (Streamlit).
- It ingests PDF resumes and CSV job records, runs multi‑agent LLM/OCR processing, builds a vector index for semantic matching, generates outreach emails, persists everything in SQLite, and visualizes via Streamlit.

##### File Structure
- Top‑level scripts represent discrete pipeline stages:
- job_data_extraction.py
- job_summary_extraction.py
- document_processing.py
- candidate_pii_extraction.py
- resume_summary_extraction.py
- resume_vector_db.py
- resume_matching.py
- email_templating.py
- 01_DashBoard.py (Streamlit UI)
- “exported_data/” holds sample Excel exports from the SQLite DB.
- requirements.txt, .gitignore, assets/ (UI assets), README.md.

##### Key Architecture From README
- **External services**: Ollama (LLM), Tesseract (OCR), Faiss (HNSW)
- **Data stores**:
- SQLite3 (`candidates.db`) for structured tables
- Faiss index files for vector embeddings
- **Tech Stack**: Python, Pandas, Streamlit

##### Main Components & Their Relationships
1. **Data Ingestion**
- Input: PDF resumes, CSV of job titles/descriptions
- Script: `job_data_extraction.py` → writes raw jobs to SQLite
2. **Job Analysis**
- Script: `job_summary_extraction.py`
- Calls: Ollama
- Writes: job key‑points back to SQLite
3. **Resume OCR & Raw Text**
- Script: `document_processing.py`
- Calls: Tesseract via Docling
- Writes: raw text path/content into SQLite
4. **PII Extraction**
- Script: `candidate_pii_extraction.py`
- Calls: Ollama
- Writes: name, email, phone to SQLite
5. **Resume Key‑Info Extraction**
- Script: `resume_summary_extraction.py`
- Calls: Ollama
- Writes: skills, summaries to SQLite
6. **Vectorization & Indexing**
- Script: `resume_vector_db.py`
- Calls: Faiss (HNSW)
- Outputs: vector index files + mapping into SQLite
7. **Matching & Scoring**
- Script: `resume_matching.py`
- Uses: vector index + job key‑points
- Calls: Ollama for detailed justifications
- Writes: top matches + scores into SQLite
8. **Email Generation**
- Script: `email_templating.py`
- Calls: Ollama
- Writes: personalized draft emails into SQLite
9. **Visualization**
- Script: `01_DashBoard.py` (Streamlit)
- Reads: SQLite + Faiss index (if needed)
- Displays: jobs, candidates, match details, emails

##### Architectural Patterns & Principles
- **Pipeline / Batch Orchestration**: discrete, idempotent stages
- **Separation of Concerns**: each Python script is a single-purpose agent
- **Data‑Centric Design**: SQLite as single source of truth, Faiss for vector store

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Data Sources
    PDFSource["PDF Resumes"]:::external
    CSVSource["CSV Job Records"]:::external

    %% Ingestion Layer
    subgraph "Ingestion"
        DataIngestion["job_data_extraction.py\n(ingest PDFs & CSV → SQLite)"]:::proc
    end

    %% Processing Layer
    subgraph "Processing"
        JobAnalysis["job_summary_extraction.py\n(Ollama)"]:::proc
        OCR["document_processing.py\n(Tesseract OCR)"]:::proc
        PII["candidate_pii_extraction.py\n(Ollama)"]:::proc
        ResumeInfo["resume_summary_extraction.py\n(Ollama)"]:::proc
        Vectorize["resume_vector_db.py\n(Faiss HNSW)"]:::proc
        Matching["resume_matching.py\n(Faiss + Ollama)"]:::proc
        EmailGen["email_templating.py\n(Ollama)"]:::proc
    end

    %% Storage Layer
    subgraph "Storage"
        SQLite["SQLite3 DB\n(candidates.db)"]:::db
        FaissIndex["Faiss Index Files"]:::db
        ExportedData["exported_data/"]:::db
    end

    %% Presentation Layer
    subgraph "Presentation"
        UI["01_DashBoard.py\n(Streamlit UI)"]:::ui
        Assets["assets/"]:::ui
    end

    %% External Services
    OllamaService["Ollama LLM"]:::external
    TesseractService["Tesseract OCR"]:::external

    %% Data Flow
    PDFSource --> DataIngestion
    CSVSource --> DataIngestion
    DataIngestion --> SQLite

    JobAnalysis --> OllamaService
    JobAnalysis --> SQLite

    OCR --> TesseractService
    OCR --> SQLite

    PII --> OllamaService
    PII --> SQLite

    ResumeInfo --> OllamaService
    ResumeInfo --> SQLite

    Vectorize --> FaissIndex
    Vectorize --> SQLite

    Matching --> FaissIndex
    Matching --> OllamaService
    Matching --> SQLite

    EmailGen --> OllamaService
    EmailGen --> SQLite

    UI --> SQLite
    UI --> FaissIndex

    %% Exported Data
    SQLite --> ExportedData

    %% Click Events
    click DataIngestion "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/job_data_extraction.py"
    click JobAnalysis "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/job_summary_extraction.py"
    click OCR "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/document_processing.py"
    click PII "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/candidate_pii_extraction.py"
    click ResumeInfo "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/resume_summary_extraction.py"
    click Vectorize "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/resume_vector_db.py"
    click Matching "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/resume_matching.py"
    click EmailGen "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/email_templating.py"
    click UI "https://github.com/stellarphantasm30/hack_the_future_hackathon/blob/main/01_DashBoard.py"
    click ExportedData "https://github.com/stellarphantasm30/hack_the_future_hackathon/tree/main/exported_data/"
    click Assets "https://github.com/stellarphantasm30/hack_the_future_hackathon/tree/main/assets/"

    %% Styles
    classDef proc fill:#d4fcb8,stroke:#4caf50,stroke-width:2px
    classDef db fill:#bbdefb,stroke:#2196f3,stroke-width:2px,stroke-dasharray: 5 5
    classDef external fill:#ffe0b2,stroke:#ff9800,stroke-width:2px,stroke-dasharray: 2 2
    classDef ui fill:#e1bee7,stroke:#9c27b0,stroke-width:2px
    classDef doc fill:#f5f5f5,stroke:#9e9e9e,stroke-width:1px,stroke-dasharray: 2 2
```
