# pyladiesams/data-ingestion-modern-stack-apr2025 architecture

> Architecture diagram of the GitHub repository pyladiesams/data-ingestion-modern-stack-apr2025, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pyladiesams/data-ingestion-modern-stack-apr2025
- Repository: https://github.com/pyladiesams/data-ingestion-modern-stack-apr2025

## Overview

##### Project type and purpose
- This is not a typical full-stack web app but a hands-on **data ingestion workshop** repository.
- Its goal is to teach how to ingest data from a REST API into a DuckDB database using the open-source Python library **dlt**, all orchestrated through Jupyter/Colab notebooks.

##### File structure
- Top-level items:
- `.gitignore`, `LICENSE`, `pyproject.toml`, `.pre-commit-config.yaml` – standard repo/config files.
- `README.md` – describes workshop, dependencies, usage.
- `workshop/` – contains the “blank” starter notebook.
- `solutions/` – contains the completed solution notebook.
- There is no separate “src” code directory: the notebook cells *are* the pipeline code.
- Dependency management is handled by `uv` (Astral’s UV tool) as specified in the README and `pyproject.toml`.

##### Key insights from the README
- The workshop uses:
- **dlt** (Data Load Tool) to materialize an ingestion pipeline.
- A REST API as the *source*.
- **DuckDB** as the *destination* database.
- Jupyter Notebook (or Google Colab) as the interactive execution environment.
- Pre-commit hooks ensure code formatting and linting before commits.

##### Main components
a. **Data Source**: REST API endpoint(s).
b. **Ingestion Engine**:
- Python environment managed by `uv`.
- Jupyter Notebook or Colab session containing code cells.
- The `dlt` library, configured in code cells.
c. **Storage**: DuckDB database file, local or in-memory.
d. **Developer Environment**:
- `uv` (dependency management)
- `pre-commit` for hooks
- Git/GitHub for version control
e. **Optional Execution Host**:
- Local machine (Jupyter Notebook)
- Remote (Google Colab)

##### Relationships and data flow
1. **Notebook (Jupyter/Colab)** → runs Python code using `dlt`.
2. **dlt** → issues HTTP requests to the **REST API**.
3. **REST API** → returns JSON or other structured data.
4. **dlt** → transforms/loads rows into **DuckDB**.
5. **DuckDB** → persists the ingested data for downstream analysis.
6. Developer can query DuckDB back in the same notebook.

##### Architectural patterns/design principles
- **Modular**: the pipeline logic is encapsulated in notebook cells; switching to a script or separate module would be trivial.
- **Dependency Isolation**: using virtual environments via `uv`.
- **Separation of Concerns**:
- Notebook = orchestration & presentation
- dlt = ingestion logic & connectors
- DuckDB = storage engine

##### Putting it all together
- `uv sync` & `pre-commit install` (setup phase)
- Execution of `dlt` code.
- Off to one side, show the “REST API” (external service).
- At the bottom, show DuckDB (local file) receiving data.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Developer Environment
    subgraph "Developer Environment"
        UV["uv sync (Dependency Management)"]:::compute
        PRE_COMMIT["pre-commit install"]:::compute
        PYPROJECT["pyproject.toml"]:::config
        PRECOMMIT_CFG[".pre-commit-config.yaml"]:::config
        README["README.md"]:::config
        GITIGNORE[".gitignore"]:::config
        LICENSE["LICENSE"]:::config
    end

    %% Execution Environment
    subgraph "Execution Environment (Local Jupyter / Colab)"
        NOTEBOOK_STARTER["Data Ingestion Notebook (Starter)"]:::compute
        NOTEBOOK_SOLUTION["Data Ingestion Notebook (Solution)"]:::compute
    end

    %% Ingestion Engine
    DLT["dlt.run() (Ingestion Engine)"]:::compute

    %% External Source
    API["REST API"]:::external

    %% Storage
    DUCKDB["DuckDB Database"]:::database

    %% Data Flow
    UV --> NOTEBOOK_STARTER
    PRE_COMMIT --> NOTEBOOK_STARTER
    UV --> NOTEBOOK_SOLUTION
    PRE_COMMIT --> NOTEBOOK_SOLUTION
    PYPROJECT -.-> NOTEBOOK_STARTER
    PYPROJECT -.-> NOTEBOOK_SOLUTION
    PRECOMMIT_CFG -.-> NOTEBOOK_STARTER
    PRECOMMIT_CFG -.-> NOTEBOOK_SOLUTION
    README --> NOTEBOOK_STARTER
    README --> NOTEBOOK_SOLUTION
    GITIGNORE --> NOTEBOOK_STARTER
    LICENSE --> NOTEBOOK_STARTER

    NOTEBOOK_STARTER -->|"executes dlt.run()"| DLT
    NOTEBOOK_SOLUTION -->|"executes dlt.run()"| DLT

    DLT -->|"HTTP GET"| API
    API -->|"JSON Response"| DLT

    DLT -->|"INSERT rows"| DUCKDB
    DUCKDB -->|"Query Data"| NOTEBOOK_STARTER
    DUCKDB -->|"Query Data"| NOTEBOOK_SOLUTION

    %% Click Events
    click NOTEBOOK_STARTER "https://github.com/pyladiesams/data-ingestion-modern-stack-apr2025/blob/main/workshop/PyLadies_2025_Open_source_open_pipelines_Data_ingestion_with_modern_data_stack.ipynb"
    click NOTEBOOK_SOLUTION "https://github.com/pyladiesams/data-ingestion-modern-stack-apr2025/blob/main/solutions/PyLadies_2025_Open_source_open_pipelines_Solutions.ipynb"
    click PYPROJECT "https://github.com/pyladiesams/data-ingestion-modern-stack-apr2025/blob/main/pyproject.toml"
    click PRECOMMIT_CFG "https://github.com/pyladiesams/data-ingestion-modern-stack-apr2025/blob/main/.pre-commit-config.yaml"
    click README "https://github.com/pyladiesams/data-ingestion-modern-stack-apr2025/blob/main/README.md"
    click GITIGNORE "https://github.com/pyladiesams/data-ingestion-modern-stack-apr2025/blob/main/.gitignore"
    click LICENSE "https://github.com/pyladiesams/data-ingestion-modern-stack-apr2025/tree/main/LICENSE"

    %% Styles
    classDef compute fill:#f0f8ff,stroke:#036,stroke-width:1px;
    classDef database fill:#e0ffe0,stroke:#060,stroke-width:1px;
    classDef external fill:#ffe0e0,stroke:#900,stroke-width:1px;
    classDef config fill:#fdf6e3,stroke:#b58900,stroke-width:1px;
```
