How ahmed-abdelwahed1/hevy-flow works
1. What this repo is
- Hevy Flow is a Python-based analytics pipeline for workout logs.
- It has two main runtime paths: an ETL job that prepares and loads data, and a Streamlit dashboard that reads the loaded data for visualization.
2. Primary data source
- The raw input is a CSV export from Hevy stored in data/workouts.csv.
- This is the ingestion boundary: everything upstream of the database starts from that local file.
3. ETL subsystem
- The etl/ package contains the data pipeline logic:
- extract.py reads and validates the CSV input.
- transform.py cleans and reshapes the workout records.
- load.py writes the processed data into the database.
- This is a staged pipeline, so data flows linearly from file ingestion to transformation to persistence.
4. Pipeline entrypoint
- main.py is the orchestration layer for the ETL job.
- It likely coordinates the three ETL phases rather than containing business logic itself, which keeps execution flow separate from transformation details.
5. Storage boundary
- The destination is Supabase, used here as a PostgreSQL-backed warehouse.
- The loader in etl/load.py is the only component that writes to this store, making it the main persistence boundary in the system.
6. Analytics/UI subsystem
- dashboard.py is the presentation layer.
- It uses Streamlit for the app runtime and Plotly for charts, so the dashboard is a consumer of the database rather than part of the ETL pipeline.
7. Configuration and runtime setup
- config.py centralizes environment-dependent settings.
- .env.example documents the required secrets/config values, and python-dotenv is used to load them at runtime.
- environment.yml defines the Conda-based Python environment, including the main data and database libraries.
8. Core dependency model
- pandas is the main in-memory data processing library across extract/transform/load.
- psycopg2-binary is the database adapter for PostgreSQL/Supabase access.
- Streamlit and Plotly are only used in the dashboard layer, not in ETL.
9. Data flow across the repo
- The end-to-end path is: CSV export → ETL extraction → cleansing/transformation → database load → dashboard reads from database.
- The dashboard does not appear to consume the CSV directly, which keeps reporting decoupled from raw ingestion.
10. Testing boundaries
- tests/ contains focused tests for extract, transform, load, and main orchestration.
- This suggests the repo treats each ETL stage as a testable unit, with separate coverage for the pipeline entrypoint.
11. CI / delivery
- .github/workflows/ci.yml indicates automated CI is in place.
- Architecturally, this supports regression checks around the ETL pipeline and its interfaces, especially useful because the system depends on data shape and database connectivity.
12. How to draw the architecture
- Draw three major zones: local raw data, ETL processing, and Supabase storage, with the dashboard as a separate consumer layer.
- Emphasize that main.py drives the batch pipeline, while dashboard.py is an independent read path into the same database.
Main entry points: extract.py extract · main.py main · dashboard.py dashboard · transform.py transform
Components 15
Raw data
Local workout export ingested by the pipeline.
- workouts.csv · Raw Hevy workout export consumed by the ETL job.
data/workouts.csv
ETL pipeline
Batch processing stages and orchestration.
- main · Entry point that coordinates the batch pipeline.
main.py - extract · Reads and validates the raw workout CSV.
etl/extract.py - transform · Cleans and reshapes workout records for loading.
etl/transform.py - load · Persists processed workout data into Supabase/PostgreSQL.
etl/load.py - pandas · Primary in-memory data processing dependency for ETL.
- Supabase · PostgreSQL-backed storage layer for processed analytics data.
Analytics UI
Read-only dashboard and its presentation stack.
- dashboard · Interactive analytics UI that reads from the database.
dashboard.py - Streamlit · Application framework used to run the dashboard.
- Plotly · Visualization library used for dashboard charts.
Ops & config
Environment, secrets, and CI support.
- config · Centralized environment-dependent settings.
config.py - dotenv · Loads secrets and configuration from environment files.
- environment.yml · Defines the Python environment and core dependencies.
environment.yml - CI workflow · Automated checks for pipeline and repository health.
.github/workflows/ci.yml - tests · Focused coverage for extraction, transformation, loading, and orchestration.
Connections 25
- workouts.csv → extract: ingests · Raw CSV is the ETL input boundary.
- main → extract: runs · Main entrypoint starts extraction.
- main → transform: runs · Main entrypoint coordinates transformation.
- main → load: runs · Main entrypoint coordinates loading.
- extract → pandas: uses · Extraction works with tabular data frames.
- extract → transform: passes data · Validated records flow into transformation.
- transform → pandas: uses · Transformation relies on pandas operations.
- transform → load: produces · Cleaned records are handed to loading.
- load → Supabase: writes · Loader persists processed data to Supabase/PostgreSQL.
- dashboard → Supabase: reads · Dashboard queries the loaded analytics store.
- dashboard → Streamlit: built on · Streamlit provides the app runtime.
- dashboard → Plotly: charts with · Plotly renders interactive visualizations.
- config → extract: configures · Shared settings affect pipeline behavior.
- config → load: configures · Connection and load settings feed the writer.
- config → dashboard: configures · Dashboard runtime uses shared environment settings.
- dotenv → config: loads · Environment variables are loaded into config.
- environment.yml → pandas: includes · Environment specification bundles core data libraries.
- environment.yml → Streamlit: includes · Environment specification includes the dashboard runtime.
- CI workflow → tests: runs · CI likely executes the repository test suite.
- CI workflow → main: checks · CI helps protect the batch pipeline entrypoint.
- CI workflow → dashboard: checks · CI helps protect the dashboard app boundary.
- tests → extract: covers · Tests validate extraction behavior.
- tests → transform: covers · Tests validate transformation behavior.
- tests → load: covers · Tests validate loading behavior.
- tests → main: covers · Tests validate pipeline orchestration.