How kwoolaid725/vacuum-fastapi-dash-etl-app works
Project Type and Purpose
• Project Type: A full-stack web application combining a FastAPI-based REST + HTML server (with OAuth2 auth), a background ETL pipeline, and an embedded Dash dashboard for live data visualization.
File Structure Highlights
• “api/” folder: FastAPI server
• main.py – ASGI app entry point
• routers/ – route modules (auth, user, test, vacuum, etc.)
• models.py, schemas.py – SQLAlchemy & Pydantic definitions
• database.py – DB session and engine (Postgres)
• oauth2.py – OAuth2 authentication
• templates/ + static/ – Jinja2 HTML templates & JS/CSS assets
• utils.py, config.py – helper functions and environment settings
• “dash_cr_test/” folder: Dash dashboard application
• app.py – Dash app instantiation, layout, callbacks
• pyspark.py – ETL logic using PySpark to read rawfiles → transform → write DB
• assets/ – Dash CSS/JS
• rawfiles under “api/rawfiles/”: source data for ETL
• postgresql-42.6.0.jar: JDBC driver (used by PySpark or direct JDBC loads)
• requirements.txt, sql.md: dependencies and SQL notes
README Insights
• Confirms dual nature: Web API + ETL + embedded Dash.
• Screenshots show login/signup, data listing pages, live charts.
Main Components
a. Client Layer
– Web Browser / Client Apps
– Interacts via HTTP/HTTPS
b. Web Server Layer
– FastAPI ASGI Server
• Authentication (OAuth2)
• REST API Endpoints (Routers)
• Server-Rendered Pages (Jinja2 Templates)
• Static File Hosting (JS/CSS)
– Embedded Dash App (mounted under FastAPI or as sub-app)
c. ETL Pipeline
– PySpark Job (dash_cr_test/pyspark.py)
– Reads “api/rawfiles/*” (CSV, XLSX)
– Transforms data (via Jupyter notebooks or scripts)
– Writes to Postgres DB (using JDBC driver jar or SQLAlchemy)
d. Data Storage
– PostgreSQL Database
– Raw file store (local filesystem under api/rawfiles/)
e. External Dependencies
– JDBC Driver (postgresql-42.6.0.jar)
– Python packages (FastAPI, SQLAlchemy, Pydantic, Dash, PySpark, Uvicorn)
Relationships and Data Flows
• Client → FastAPI:
• GET/POST web pages (HTML forms)
• AJAX/REST calls (JSON)
• FastAPI → Database:
• SQLAlchemy sessions for CRUD
• OAuth2 token store
• ETL → Database:
• Batch load/transform jobs run periodically or on-demand
• Dash App → Database:
• Reads processed data for visualizations (via SQLAlchemy or JDBC)
• Static Assets & Templates served by FastAPI directly to Client
Architectural Patterns
• Modular Router pattern (FastAPI routers for separation of concerns)
• MVC-like layering:
• Models (models.py)
• Views (Jinja2 templates, Dash layouts)
• Controllers (router endpoints + callback functions)
• Embedded Single-Page/Dash Application within main ASGI app
• ETL as a separate module but sharing the same DB