# datatalksclub/llm-zoomcamp architecture

> Architecture diagram of the GitHub repository datatalksclub/llm-zoomcamp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datatalksclub/llm-zoomcamp
- Repository: https://github.com/datatalksclub/llm-zoomcamp

## Overview

##### Project Type and Purpose
- This is not a production codebase but an educational “course repository” for the LLM Zoomcamp. It guides students through building Retrieval‐Augmented Generation (RAG) systems in Python, covering:
- Introduction to LLMs and RAG (Elasticsearch, OpenAI API)
- Vector search (embeddings, vector DBs)
- Evaluation (search/LLM evaluation)
- Monitoring (dashboards, online evaluation)
- Best practices (hybrid search, reranking)
- A full end‐to‐end example project

##### File Structure
- Top‐level directories “01-intro” through “06-project-example” correspond to modules, each containing scripts/notebooks, data, and READMEs.
- The “cohorts” directory houses hands‐on lab code, Docker configurations, and data for in‐class exercises.
- Supporting files include:
- `docker-compose.yaml`, `Dockerfile` (for reproducible environments)
- `.ipynb` notebooks (illustrating pipeline steps interactively)
- JSON/CSV data artifacts (documents, embeddings, ground‐truth)
- Monitoring configs (Grafana JSON, dashboards)

##### Core System Architecture for a RAG Pipeline
A. Content Ingestion & Preprocessing
– Source documents (PDFs, JSON, CSV)
– Preprocessor component (tokenization, cleaning)
– Output: embeddings + metadata
B. Indexing & Storage Layer
– Elasticsearch (text search)
– Vector database (e.g., FAISS, Pinecone)
C. Retrieval & Orchestration Service
– API / Python orchestrator (LangChain or custom)
– Query dispatcher:
- Keyword search → ES
- Vector query → Vector DB
– Results aggregator
D. LLM Inference Engine
– OpenAI API / HF inference service
– Prompts with retrieved context → generates answers
E. Evaluation Pipeline
– Offline evaluator (notebooks scoring search and LLM outputs)
– Online evaluator (LLM-as-judge, user feedback)
– Ground-truth comparator (CSV of Q&A)
F. Monitoring & Dashboarding
– Metrics collector (logging API calls, latencies, accuracy)
– Grafana (dashboard JSON, alerts)
G. User Interface (optional)
– Jupyter demos or minimal web UI (Streamlit/Flask)

##### Relationships and Data Flow
1. Documents → Preprocessor → Indexes (ES & Vector DB)
2. User Query → Orchestrator → Retrieval Services → Top-K contexts
3. Context + Query → LLM API → Answer
4. Answer → Evaluation Module → Metrics
5. Metrics → Monitoring Dashboard
- Bi-directional feedback from Monitoring back to Best Practices (e.g., retraining, prompt tuning).

##### Architectural Patterns & Design Principles
- Pipeline pattern: clearly separate ingestion, retrieval, inference, evaluation, monitoring.
- Modular microservices (or Python modules) for each step—easy to swap components (e.g., replace ES with Pinecone).
- Configuration‐as‐code (Docker/Docker Compose, JSON dashboards).
- Reproducibility via notebooks and versioned data artifacts.

##### Relevant Technologies / Libraries
- Python + Jupyter Notebooks
- Elasticsearch (text search)
- FAISS / other vector store
- OpenAI API / Hugging Face models
- LangChain or custom orchestrator scripts
- Docker, Docker Compose
- Grafana for monitoring
- Pandas/NumPy for evaluation scripts

## Diagram (Mermaid)

```mermaid
graph TB
    %% Module 1: Intro & ES
    subgraph "Module 1 (Intro & ES)"
        direction TB
        RawDocs["Raw Documents<br/>documents.json<br/>documents-llm.json"]:::data
        click RawDocs "https://github.com/datatalksclub/llm-zoomcamp/blob/main/01-intro/documents.json"
        click RawDocs "https://github.com/datatalksclub/llm-zoomcamp/blob/main/01-intro/documents-llm.json"
        ParseNotebook["Preprocessor Notebook<br/>parse-faq.ipynb"]:::service
        click ParseNotebook "https://github.com/datatalksclub/llm-zoomcamp/blob/main/01-intro/parse-faq.ipynb"
        PreprocessScript["Preprocessor Script<br/>minsearch.py"]:::service
        click PreprocessScript "https://github.com/datatalksclub/llm-zoomcamp/blob/main/01-intro/minsearch.py"
        ESSetup["Elasticsearch Setup & Demo"]:::infra
        click ESSetup "https://github.com/datatalksclub/llm-zoomcamp/blob/main/01-intro/elastic-search.md"
    end

    %% Module 2: Vector Search
    subgraph "Module 2 (Vector Search)"
        direction TB
        VectorModule["Vector Search Module"]:::service
        click VectorModule "https://github.com/datatalksclub/llm-zoomcamp/blob/main/02-vector-search/README.md"
        VectorDB["Vector Database"]:::data
    end

    %% Module 3: Evaluation
    subgraph "Module 3 (Evaluation)"
        direction TB
        EvalNotebookOffline["Offline Evaluator Notebook"]:::eval
        click EvalNotebookOffline "https://github.com/datatalksclub/llm-zoomcamp/blob/main/03-evaluation/README.md"
        GroundTruthCSV["Ground-Truth Comparator<br/>ground-truth-data.csv"]:::data
        click GroundTruthCSV "https://github.com/datatalksclub/llm-zoomcamp/blob/main/05-best-practices/ground-truth-data.csv"
        EvalNotebookOnline["Online Evaluator (LLM-as-Judge)"]:::eval
        click EvalNotebookOnline "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/04-monitoring/offline-rag-evaluation.ipynb"
    end

    %% Module 4: Monitoring
    subgraph "Module 4 (Monitoring)"
        direction TB
        MetricsCollector["Metrics Collector"]:::infra
        MonitoringDashboard["Grafana Dashboard"]:::infra
        click MonitoringDashboard "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/04-monitoring/dashboard.json"
        MonitoringApp["Monitoring Service App"]:::service
        click MonitoringApp "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/04-monitoring/app/app.py"
        DBHelpers["Monitoring DB Helpers"]:::service
        click DBHelpers "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/04-monitoring/app/db.py"
        DataGen["Monitoring Data Generators"]:::service
        click DataGen "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/04-monitoring/app/generate_data.py"
    end

    %% Module 5: Best Practices
    subgraph "Module 5 (Best Practices)"
        direction TB
        HybridES["Hybrid Search & Reranking ES"]:::service
        click HybridES "https://github.com/datatalksclub/llm-zoomcamp/blob/main/05-best-practices/hybrid-search-and-reranking-es.ipynb"
        HybridLC["Hybrid Search & Reranking LangChain"]:::service
        click HybridLC "https://github.com/datatalksclub/llm-zoomcamp/blob/main/05-best-practices/hybrid-search-langchain.ipynb"
        PracDocs["Best Practices Documents"]:::data
        click PracDocs "https://github.com/datatalksclub/llm-zoomcamp/blob/main/05-best-practices/documents-with-ids.json"
    end

    %% Module 6: End-To-End Project
    subgraph "Module 6 (End-to-End Project)"
        direction TB
        ProjectSummary["Project Example Summary"]:::service
        click ProjectSummary "https://github.com/datatalksclub/llm-zoomcamp/blob/main/06-project-example/content-processing-summary.md"
    end

    %% Cohort Labs
    subgraph "Cohort Labs & Orchestration"
        direction TB
        Orchestrator["Retrieval Orchestration Service"]:::service
        click Orchestrator "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/05-orchestration/code/06_retrieval.py"
        HF1["HuggingFace Flan-T5 Demo"]:::service
        click HF1 "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/02-open-source/huggingface-flan-t5.ipynb"
        HF2["HuggingFace Mistral-7B Demo"]:::service
        click HF2 "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/02-open-source/huggingface-mistral-7b.ipynb"
        DockerCompose["Docker Compose Config"]:::infra
        click DockerCompose "https://github.com/datatalksclub/llm-zoomcamp/blob/main/cohorts/2024/02-open-source/docker-compose.yaml"
    end

    %% Data Layer connections
    RawDocs -->|ingest| ParseNotebook
    ParseNotebook -->|script| PreprocessScript
    PreprocessScript -->|"embeddings & metadata"| ESSetup
    PreprocessScript -->|"embeddings"| VectorDB

    %% Retrieval & Inference
    Orchestrator -->|keyword query| ESSetup
    Orchestrator -->|vector query| VectorDB
    ESSetup -->|top-K docs| Orchestrator
    VectorDB -->|top-K docs| Orchestrator

    Orchestrator -->|context + query| HF1
    Orchestrator -->|context + query| HF2

    %% Evaluation flows
    HF1 -->|answers| EvalNotebookOffline
    HF2 -->|answers| EvalNotebookOffline
    GroundTruthCSV -->|compare| EvalNotebookOffline
    EvalNotebookOffline -->|scores| MetricsCollector
    EvalNotebookOnline -->|feedback| MetricsCollector

    %% Monitoring flows
    MetricsCollector -->|metrics| MonitoringDashboard
    MonitoringDashboard -->|alerts| DataGen
    DataGen -->|simulated events| MonitoringApp
    MonitoringApp -->|logs| DBHelpers
    DBHelpers -->|data| MetricsCollector

    %% Best practices loop
    HybridES -->|results| EvalNotebookOffline
    HybridLC -->|results| EvalNotebookOffline

    %% End-to-end summary
    ProjectSummary --> RawDocs
    ProjectSummary --> Orchestrator
    ProjectSummary --> MonitoringDashboard

    %% Course README
    CourseReadme["Course README & Intro"]:::service
    click CourseReadme "https://github.com/datatalksclub/llm-zoomcamp/blob/main/README.md"
    CourseReadme -->|overview| RawDocs

    %% Styles
    classDef data fill:#D0E8FF,stroke:#0366d6,color:#0366d6;
    classDef service fill:#DFFFE0,stroke:#2E8B57,color:#2E8B57;
    classDef eval fill:#FFE8B0,stroke:#DAA520,color:#DAA520;
    classDef infra fill:#F0F0F0,stroke:#808080,color:#333333;
```
