# kwoolaid725/vacuum-fastapi-dash-etl-app architecture

> Architecture diagram of the GitHub repository kwoolaid725/vacuum-fastapi-dash-etl-app, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kwoolaid725/vacuum-fastapi-dash-etl-app
- Repository: https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app

## Overview

##### Project Type and Purpose
- Project Type: A full-stack web application combining a FastAPI-based REST + HTML server (with OAuth2 auth), a background ETL pipeline, and an embedded Dash dashboard for live data visualization.

##### File Structure Highlights
- “api/” folder: FastAPI server
- main.py – ASGI app entry point
- routers/ – route modules (auth, user, test, vacuum, etc.)
- models.py, schemas.py – SQLAlchemy & Pydantic definitions
- database.py – DB session and engine (Postgres)
- oauth2.py – OAuth2 authentication
- templates/ + static/ – Jinja2 HTML templates & JS/CSS assets
- utils.py, config.py – helper functions and environment settings
- “dash_cr_test/” folder: Dash dashboard application
- app.py – Dash app instantiation, layout, callbacks
- pyspark.py – ETL logic using PySpark to read rawfiles → transform → write DB
- assets/ – Dash CSS/JS
- rawfiles under “api/rawfiles/”: source data for ETL
- postgresql-42.6.0.jar: JDBC driver (used by PySpark or direct JDBC loads)
- requirements.txt, sql.md: dependencies and SQL notes

##### README Insights
- Confirms dual nature: Web API + ETL + embedded Dash.
- Screenshots show login/signup, data listing pages, live charts.

##### Main Components
a. Client Layer
– Web Browser / Client Apps
– Interacts via HTTP/HTTPS
b. Web Server Layer
– FastAPI ASGI Server
- Authentication (OAuth2)
- REST API Endpoints (Routers)
- Server-Rendered Pages (Jinja2 Templates)
- Static File Hosting (JS/CSS)
– Embedded Dash App (mounted under FastAPI or as sub-app)
c. ETL Pipeline
– PySpark Job (dash_cr_test/pyspark.py)
– Reads “api/rawfiles/*” (CSV, XLSX)
– Transforms data (via Jupyter notebooks or scripts)
– Writes to Postgres DB (using JDBC driver jar or SQLAlchemy)
d. Data Storage
– PostgreSQL Database
– Raw file store (local filesystem under api/rawfiles/)
e. External Dependencies
– JDBC Driver (postgresql-42.6.0.jar)
– Python packages (FastAPI, SQLAlchemy, Pydantic, Dash, PySpark, Uvicorn)

##### Relationships and Data Flows
- Client → FastAPI:
- GET/POST web pages (HTML forms)
- AJAX/REST calls (JSON)
- FastAPI → Database:
- SQLAlchemy sessions for CRUD
- OAuth2 token store
- ETL → Database:
- Batch load/transform jobs run periodically or on-demand
- Dash App → Database:
- Reads processed data for visualizations (via SQLAlchemy or JDBC)
- Static Assets & Templates served by FastAPI directly to Client

##### Architectural Patterns
- Modular Router pattern (FastAPI routers for separation of concerns)
- MVC-like layering:
- Models (models.py)
- Views (Jinja2 templates, Dash layouts)
- Controllers (router endpoints + callback functions)
- Embedded Single-Page/Dash Application within main ASGI app
- ETL as a separate module but sharing the same DB

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    subgraph "Client Layer"
        direction TB
        Browser("Web Browser / Client"):::client
    end

    %% FastAPI Layer
    subgraph "FastAPI Server" 
        direction TB
        Main["main.py"]:::service
        Routers["routers/ (Auth, User, Test, Vacuum)"]:::service
        Models["models.py"]:::service
        Schemas["schemas.py"]:::service
        DatabaseConfig["database.py"]:::service
        OAuth2["oauth2.py"]:::service
        Config["config.py"]:::service
        Utils["utils.py"]:::service
        Templates["templates/"]:::file
        Static["static/"]:::file
    end

    %% Dash + ETL Layer
    subgraph "Dash & ETL Module"
        direction TB
        DashApp["app.py (Dash Dashboard)"]:::service
        ETL["pyspark.py (ETL Pipeline)"]:::service
        DashAssets["assets/"]:::file
    end

    %% Data Storage
    subgraph "Data Storage"
        direction TB
        RawFiles["rawfiles/ (CSV, XLSX)"]:::file
        JDBCJar["postgresql-42.6.0.jar"]:::ext
        Postgres[(PostgreSQL Database)]:::db
    end

    %% External Dependencies
    Requirements["requirements.txt"]:::ext
    SQLDocs["sql.md"]:::ext

    %% Connections - Real-time flows
    Browser -->|HTTP/HTTPS| Main
    Main -->|includes routers| Routers
    Main -->|loads models/schemas| Models
    Main -->|"uses ORM"| Schemas
    Routers -->|DB session| DatabaseConfig
    Routers -->|auth flows| OAuth2
    Routers -->|config/env| Config
    Routers -->|helper funcs| Utils
    Main -->|"serves"| Templates
    Main -->|"serves"| Static

    %% Dash embedded
    Main -->|mounts Dash| DashApp
    DashApp -->|reads data| Postgres

    %% ETL flows
    RawFiles ---|batch read| ETL
    ETL ---|batch write| Postgres
    JDBCJar -.->|JDBC driver| ETL

    %% Direct DB access
    DatabaseConfig -->|engine/session| Postgres

    %% External deps
    Requirements -.-> Main
    Requirements -.-> DashApp
    Requirements -.-> ETL
    SQLDocs -.-> Models

    %% Click Events
    click Main "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/api/main.py"
    click Routers "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/tree/master/api/routers/"
    click Models "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/api/models.py"
    click Schemas "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/api/schemas.py"
    click DatabaseConfig "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/api/database.py"
    click OAuth2 "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/api/oauth2.py"
    click Config "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/api/config.py"
    click Utils "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/api/utils.py"
    click Templates "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/tree/master/api/templates/"
    click Static "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/tree/master/api/static/"
    click RawFiles "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/tree/master/api/rawfiles/"
    click DashApp "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/dash_cr_test/app.py"
    click ETL "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/dash_cr_test/pyspark.py"
    click DashAssets "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/tree/master/dash_cr_test/assets/"
    click JDBCJar "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/postgresql-42.6.0.jar"
    click Requirements "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/requirements.txt"
    click SQLDocs "https://github.com/kwoolaid725/vacuum-fastapi-dash-etl-app/blob/master/sql.md"

    %% Styles
    classDef client fill:#B3E5FC,stroke:#0288D1;
    classDef service fill:#BBDEFB,stroke:#0D47A1;
    classDef db fill:#FFE0B2,stroke:#E65100;
    classDef file fill:#E1BEE7,stroke:#6A1B9A;
    classDef ext fill:#F0F4C3,stroke:#827717;
```
