# ed-donner/sds architecture

> Architecture diagram of the GitHub repository ed-donner/sds, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ed-donner/sds
- Repository: https://github.com/ed-donner/sds

## Overview

##### Project type and purpose
- This repo is not a traditional full-stack application but rather a curriculum repository for an 8-week “SuperDataScience AI Bootcamp.” It contains:
- Course materials (Jupyter notebooks under `week1/…`, `cohort1/week1/…`, etc.)
- Knowledge-base markdown files (in `knowledge-base/` folders)
- Python scripts for labs, ingestion, evaluation, and agentic-AI demos
- Setup and troubleshooting guides
- Its “system” is essentially a content-and-code delivery platform for students, featuring sample pipelines that call OpenAI APIs.

##### File structure
Top-level:
- `cohort1/`, `week1/`, `week2/`, … — multiple parallel folder structures holding weekly materials.
- `guides/` — standalone Jupyter-based tutorials on Git, Python, Async, etc.
- `setup/` — environment installation and troubleshooting scripts.
- `assets/` — static images (e.g., banner).
- `pyproject.toml`, `uv.lock`, `.python‐version` — project configuration and dependencies.
Within each week or cohort week folder:
- `lab*.ipynb` or `.py` — hands-on exercises
- `knowledge-base/` — markdown content for company, contracts, employees, products
- Supporting Python modules: e.g., week2’s `ingest.py`, `embeddings.py`, `evaluator.py`, week3’s deep_research agents.

##### Extract architecture hints from the README
- Purpose: guide through LLMs, Agents, and Production AI systems.
- It mentions “making API calls to the OpenAI API,” “deploying AI systems for Production,” and points students at setup and troubleshooting.
- Ingest markdown “knowledge bases”
- Generate embeddings
- Query LLMs
- Evaluate results
- Orchestrate multi-agent workflows.

##### Main components
a. Content & Code Repositories
- Knowledge-base files (markdown)
- Lab notebooks (Jupyter)
- Python modules (ingest, embeddings, evaluator, agent scripts)
- Guides & setup docs
b. Execution Environment
- Student workstation/IDE (VS Code with Cursor)
- Python runtime + venv (managed by `pyproject.toml` and `.python-version`)
- Jupyter Server
c. AI Pipeline Subsystems
- Ingestion service (scripts read markdown)
- Embedding generator (calls OpenAI embeddings API)
- LLM query service (calls Chat/Completion endpoints)
- Evaluation engine (tests against `tests.jsonl`)
- Multi-Agent orchestrator (deep_research agents)
d. External Services
- OpenAI API
- (Optionally) Vector store or local cache for embeddings
- GitHub for version control

##### Relationships & data flow
- Student triggers a lab notebook →
1. Notebook imports Python modules (`ingest.py`, `embeddings.py`, etc.)
2. Ingest reads markdown files → passes raw text →
3. Embedding generator sends text to OpenAI embeddings endpoint → receives vectors →
4. Notebook or evaluation script issues queries to LLM via OpenAI completions API →
5. Evaluation module compares LLM output against ground-truth (`tests.jsonl`) → feedback.
- In week3 deep_research: Planner Agent → Research Manager → Search Agent / Push Agent / Writer Agent loop.

##### Architectural patterns & design principles
- Modular: each pipeline stage is its own Python module or notebook.
- Separation of concerns: ingestion, embedding, querying, evaluation all decoupled.
- Agent orchestration: follows “manager–worker” pattern for multi-agent experiments.
- Configuration as code: dependency management via `pyproject.toml` and lockfiles.
- Content as data: knowledge-base markdown separated from code.

##### Detail
- Don’t just lump “labs” together—break out major pipelines week by week.
- Multi-agent workflow separately.
- Configuration files (`pyproject.toml`, `.python-version`) feeding into the environment layer.

- Precisely mirrors this bootcamp’s repo structure and content-code separation
- Illustrates the AI pipelines students implement each week
- Displays interactions with external APIs and environment setup

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Environment Layer
    subgraph "Environment Layer"
        direction TB
        IDE["Student IDE (VS Code)"]:::env
        PYTHON["Python Runtime + venv"]:::env
        JUPYTER["Jupyter Server"]:::env
        CONFIG["pyproject.toml\n.python-version\nuv.lock"]:::env
    end

    %% Content Layer
    subgraph "Content Layer"
        direction TB
        subgraph "Knowledge-Base"
            KB1["Cohort1 Week1 Knowledge-Base"]:::content
            KB2["Week2 Knowledge-Base"]:::content
        end
        subgraph "Lab Notebooks"
            LN1["cohort1/week1/lab1.ipynb"]:::content
            LN2["week1/lab1.ipynb"]:::content
            LN3["week2/lab1.ipynb"]:::content
        end
        GUIDES["Guides & Tutorials"]:::content
        ASSETS["Static Assets (images)"]:::content
    end

    %% Code Layer
    subgraph "Code Layer"
        direction TB
        subgraph "Week2 Pipeline"
            direction TB
            INGEST["ingest.py"]:::code
            EMBED["embeddings.py"]:::code
            EVAL["evaluator.py"]:::code
            TESTS["tests.jsonl"]:::code
        end
        subgraph "Week3 Multi-Agent"
            direction TB
            DR["deep_research.py"]:::code
            PLAN["planner_agent.py"]:::code
            MANAGER["research_manager.py"]:::code
            SEARCH["search_agent.py"]:::code
            PUSH["push_agent.py"]:::code
            WRITE["writer_agent.py"]:::code
        end
        subgraph "Week4 Digital Twin"
            direction TB
            TWIN_APP["app.py"]:::code
            TWIN_DT["digital_twin.py"]:::code
            TWIN_CTX["context.py"]:::code
            TWIN_CTS["contacts.py"]:::code
            TWIN_RES["resources.py"]:::code
            TWIN_STY["styling.py"]:::code
            TWIN_ADM["admin.py"]:::code
            TWIN_MCP["mcp_servers.py"]:::code
            TWIN_PUSH["push.py"]:::code
            VALID4["validate.py"]:::code
        end
        subgraph "Setup & Troubleshooting"
            direction TB
            SETUP1["SETUP-PC.md"]:::code
            SETUP2["troubleshooting.ipynb"]:::code
            DIAG["diagnostics.py"]:::code
        end
    end

    %% External Services
    subgraph "External Services"
        direction TB
        OPENAI["OpenAI API"]:::external
        GITHUB["GitHub"]:::external
    end

    %% Data Flows
    IDE -->|opens| LN1
    IDE -->|opens| LN2
    IDE -->|opens| LN3
    LN1 -->|imports| INGEST
    LN2 -->|imports| EMBED
    LN3 -->|imports| EVAL
    INGEST -->|reads| KB1
    EMBED -->|reads| KB2
    EMBED -->|calls embeddings API| OPENAI
    EVAL -->|reads ground truth| TESTS
    LN3 -->|queries| OPENAI
    EVAL -->|feedback| LN3
    DR -->|orchestrates| PLAN
    PLAN -->|delegates| MANAGER
    MANAGER -->|spawns| SEARCH
    MANAGER -->|spawns| PUSH
    MANAGER -->|spawns| WRITE
    TWIN_APP -->|uses| TWIN_DT
    TWIN_APP -->|uses| TWIN_CTX
    TWIN_APP -->|uses| TWIN_CTS
    TWIN_APP -->|uses| TWIN_RES
    TWIN_APP -->|uses| TWIN_STY
    TWIN_APP -->|uses| TWIN_ADM
    TWIN_APP -->|uses| TWIN_MCP
    TWIN_APP -->|uses| TWIN_PUSH
    TWIN_APP -->|validates| VALID4
    SETUP1 -->|assists| IDE
    SETUP2 -->|assists| IDE
    DIAG -->|diagnoses| PYTHON
    CONFIG -->|configures| PYTHON
    PYTHON -->|runs| JUPYTER
    JUPYTER -->|hosts| LN1
    GITHUB -->|version control| KB1
    GITHUB -->|version control| INGEST

    %% Click Events
    click KB1 "https://github.com/ed-donner/sds/tree/main/cohort1/week1/knowledge-base/"
    click KB2 "https://github.com/ed-donner/sds/tree/main/week2/knowledge-base/"
    click LN1 "https://github.com/ed-donner/sds/blob/main/cohort1/week1/lab1.ipynb"
    click LN2 "https://github.com/ed-donner/sds/blob/main/week1/lab1.ipynb"
    click LN3 "https://github.com/ed-donner/sds/blob/main/week2/lab1.ipynb"
    click INGEST "https://github.com/ed-donner/sds/blob/main/week2/ingest.py"
    click EMBED "https://github.com/ed-donner/sds/blob/main/week2/embeddings.py"
    click EVAL "https://github.com/ed-donner/sds/blob/main/week2/evaluator.py"
    click TESTS "https://github.com/ed-donner/sds/blob/main/week2/tests.jsonl"
    click DR "https://github.com/ed-donner/sds/blob/main/cohort1/week3/deep_research/deep_research.py"
    click PLAN "https://github.com/ed-donner/sds/blob/main/cohort1/week3/deep_research/planner_agent.py"
    click MANAGER "https://github.com/ed-donner/sds/blob/main/cohort1/week3/deep_research/research_manager.py"
    click SEARCH "https://github.com/ed-donner/sds/blob/main/cohort1/week3/deep_research/search_agent.py"
    click PUSH "https://github.com/ed-donner/sds/blob/main/cohort1/week3/deep_research/push_agent.py"
    click WRITE "https://github.com/ed-donner/sds/blob/main/cohort1/week3/deep_research/writer_agent.py"
    click TWIN_APP "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/app.py"
    click TWIN_DT "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/digital_twin.py"
    click TWIN_CTX "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/context.py"
    click TWIN_CTS "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/contacts.py"
    click TWIN_RES "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/resources.py"
    click TWIN_STY "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/styling.py"
    click TWIN_ADM "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/admin.py"
    click TWIN_MCP "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/mcp_servers.py"
    click TWIN_PUSH "https://github.com/ed-donner/sds/blob/main/cohort1/week4/twin/push.py"
    click SETUP1 "https://github.com/ed-donner/sds/blob/main/setup/SETUP-PC.md"
    click SETUP2 "https://github.com/ed-donner/sds/blob/main/setup/troubleshooting.ipynb"
    click DIAG "https://github.com/ed-donner/sds/blob/main/setup/diagnostics.py"
    click CONFIG "https://github.com/ed-donner/sds/blob/main/pyproject.toml"

    %% Styles
    classDef content fill:#D0E7FF,stroke:#0366D6,color:#0366D6
    classDef code fill:#E6FFEA,stroke:#28A745,color:#28A745
    classDef env fill:#FFF4E5,stroke:#D69E2E,color:#D69E2E
    classDef external fill:#FFE3E3,stroke:#CB2431,color:#CB2431
```
