# calebagoha123/jonathan_ta architecture

> Architecture diagram of the GitHub repository calebagoha123/jonathan_ta, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/calebagoha123/jonathan_ta
- Repository: https://github.com/calebagoha123/jonathan_ta

## Overview

##### Main Components
a. Document Processing Pipeline
- Source PDFs (in src/web/static/docs/)
- document_loader.py (ingests raw PDFs)
- preprocessor.py (parses, text-chunks, attaches metadata)
b. Vector Storage
- ChromaDB instance (data/chroma_db/)
- Collections per document type
- chroma.sqlite3 for persistence
c. Embedding & Retrieval Layer
- Sentence-Transformer model (`all-MiniLM-L6-v2`)
- Embeddings module (src/rag/embeddings.py)
- Retriever (src/rag/retriever.py)
- Semantic search + metadata filtering
d. LLM & Prompting
- External LLM endpoint (gpt-4o-mini)
- Query formatting / prompt assembly
e. Web Interface
- Streamlit app (src/web/streamlit_app.py)
- Templates (src/web/templates/index.html)
- Static assets & course PDFs
- Session/context management

##### Architectural Patterns
- Layered architecture: Ingestion → Data store → Retrieval/Compute → Presentation
- RAG pattern: Retrieval augmented generation with vector DB + LLM
- Modularity: Separate concerns into src/data, src/rag, src/web

##### Technologies & Frameworks
- PDF parsing & preprocessing: PyPDF2 or similar (in document_loader.py)
- Chunking logic (1000 tokens/100 overlap)
- Vector DB: ChromaDB (local embedded)
- Embeddings: Sentence Transformers (`all-MiniLM-L6-v2`)
- LLM: OpenAI gpt-4o-mini via API
- Web: Streamlit, Jinja2 (templates), CSS/HTML static assets
- Dev environment: Docker devcontainer

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        SourcePDFs["Source PDFs\n(src/web/static/docs/)"]:::doc
        Loader["document_loader.py"]:::compute
        Preprocessor["preprocessor.py"]:::compute
    end

    %% Storage Layer
    subgraph "Storage Layer"
        ChromaDir["ChromaDB Directory\n(data/chroma_db/)"]:::storage
        SQLite["chroma.sqlite3\n(data/chroma_db/chroma.sqlite3)"]:::storage
    end

    %% Retrieval & Compute Layer
    subgraph "Retrieval & Compute Layer"
        Embeddings["embeddings.py"]:::compute
        Retriever["retriever.py"]:::compute
        LLM["External LLM\n(gpt-4o-mini)"]:::external
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        StreamlitApp["streamlit_app.py"]:::ui
        Template["index.html"]:::ui
        StaticDocs["Static PDFs\n(src/web/static/docs/)"]:::doc
    end

    %% Infrastructure Layer
    subgraph "Infrastructure"
        Devcontainer["devcontainer.json\n(.devcontainer/devcontainer.json)"]:::env
        Requirements["requirements.txt"]:::env
    end

    %% Data Flows
    SourcePDFs -->|"Raw PDFs"| Loader
    Loader -->|"Extracted text chunks + metadata"| Preprocessor
    Preprocessor -->|"Vectors"| Embeddings
    Embeddings -->|"Store vectors"| ChromaDir
    ChromaDir -->|"Persistence"| SQLite

    StreamlitApp -->|"User query"| Retriever
    Retriever -->|"Vector search"| ChromaDir
    ChromaDir -->|"Top-k vectors"| Retriever
    Retriever -->|"Top chunks + query"| LLM
    LLM -->|"Response JSON"| StreamlitApp

    %% Click Events
    click Loader "https://github.com/calebagoha123/jonathan_ta/blob/main/src/data/document_loader.py"
    click Preprocessor "https://github.com/calebagoha123/jonathan_ta/blob/main/src/data/preprocessor.py"
    click SourcePDFs "https://github.com/calebagoha123/jonathan_ta/tree/main/src/web/static/docs/"
    click ChromaDir "https://github.com/calebagoha123/jonathan_ta/tree/main/data/chroma_db/"
    click SQLite "https://github.com/calebagoha123/jonathan_ta/blob/main/data/chroma_db/chroma.sqlite3"
    click Embeddings "https://github.com/calebagoha123/jonathan_ta/blob/main/src/rag/embeddings.py"
    click Retriever "https://github.com/calebagoha123/jonathan_ta/blob/main/src/rag/retriever.py"
    click StreamlitApp "https://github.com/calebagoha123/jonathan_ta/blob/main/src/web/streamlit_app.py"
    click Template "https://github.com/calebagoha123/jonathan_ta/blob/main/src/web/templates/index.html"
    click Devcontainer "https://github.com/calebagoha123/jonathan_ta/blob/main/.devcontainer/devcontainer.json"
    click Requirements "https://github.com/calebagoha123/jonathan_ta/blob/main/requirements.txt"

    %% Styles
    classDef compute fill:#bbdefb,stroke:#1e88e5;
    classDef storage fill:#c8e6c9,stroke:#388e3c;
    classDef ui fill:#ffcdd2,stroke:#e53935;
    classDef doc fill:#fff9c4,stroke:#fbc02d;
    classDef external fill:#d1c4e9,stroke:#5e35b1;
    classDef env fill:#ffe0b2,stroke:#fb8c00;
```
