# modelearth/profile architecture

> Architecture diagram of the GitHub repository modelearth/profile, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/modelearth/profile
- Repository: https://github.com/modelearth/profile

## Overview

##### Project type and purpose
- This is a monorepo for a static‐site–driven “Profile Tools” portal (model.earth/profile). It bundles:
- Multiple interactive visualization modules (charts/d3, charts/echarts, charts/sankey)
- A React/Vite “food-nutrition-app” under item/food-nutrition-app
- Data‐prep and ETL scripts (Python, Jupyter notebooks, DuckDB, Supabase) in prep/, imp­acts/, trade/, salesforce/, crm/, databricks/
- Hosting/deployment scripts targeting Azure static hosting (azure/azure.sh)

##### File structure
- Top‐level “azure/” contains deployment scripts and an HTML landing page for Azure Static Web Apps.
- “charts/” holds several self-contained D3/Chord, ECharts, and Sankey visualization demos.
- “item/food-nutrition-app” is a standalone React app (src/, public/, vite.config.js).
- “prep/”, “impacts/”, “trade/”, “salesforce/”, “crm/”, “databricks/” each contain data ingestion or transformation code (Python scripts, Jupyter notebooks, SQL loaders, DuckDB and Supabase loaders).
- Data files (CSV, JSON, Parquet) sit alongside their loader code or under impacts/json, impacts/parquet, trade/naics, etc.
- No monolithic backend—data pipelines produce static artifacts or load into lightweight DBs (DuckDB, Supabase).

##### README
- The only README at root simply brands this as “Profile Tools” and points to the online portal. All architectural details must be inferred from file tree.

a. Main components
1. External Data Sources
– Exiobase CSVs, NAICS spreadsheets, trade YAML, Salesforce directory data
2. ETL / Data Processing Layer
– Python scripts & Jupyter notebooks under prep/, impacts/, trade/, crm/, salesforce/
– Databricks notebooks & DuckDB loaders, Supabase loaders
3. Data Storage
– Local flat files (CSV, JSON, Parquet)
– DuckDB database files (.duckdb)
– Supabase managed PostgreSQL
4. Static Site Generator / Build
– “azure/azure.sh” deployment script
– Static HTML pages under each module (index.html, charts/*/*.html, etc.)
– React/Vite build for “food-nutrition-app”
5. Frontend Presentation Layer
– Static HTML/CSS/JS modules (D3, ECharts, Sankey)
– React app served from item/food-nutrition-app
6. Hosting and CDN
– Azure Static Web Apps
– CDN edge cache
7. External Services
– Supabase (optional runtime data queries)
– Salesforce API for directory updates
– Databricks compute environment (optional)

b. Relationships and data flows
- Hosting serves to end-user browsers. End-user browsers load frontends which may optionally call Supabase or DuckDB web endpoints.
- Salesforce directory update scripts feed into ETL and then potentially update Supabase or flat files.

c. Architectural patterns and design principles
- Micro-frontend style: each chart or demo lives in its own folder with its assets and data.
- “Jamstack” static‐first approach: precompute and serve artifacts; minimize runtime servers.
- Data pipelines are loosely coupled: ETL scripts produce files or DB tables that the site picks up at build time.

d. Key technologies and libraries
- Frontend: HTML/CSS/JS, D3.js, ECharts, Sankey, React + Vite
- Data Processing: Python (pandas, DuckDB), Jupyter, Databricks notebooks, Supabase Python clients
- Storage: flat files (CSV/JSON/Parquet), DuckDB, Supabase Postgres
- Deployment: Azure Static Web Apps, azure CLI (azure.sh)

##### Detail
- Directory’s role as a mini-component.
- How data formats (CSV, JSON, Parquet) feed into visualizations.
- Do not overcomplicate: treat each folder with its own HTML/JS as a self-contained micro-frontend.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "External Data Sources"
        EDS1["Exiobase CSVs"]:::external
        EDS2["NAICS Excel"]:::external
        EDS3["Trade JSON/YAML"]:::external
        EDS4["Salesforce Directory"]:::external
    end

    subgraph "ETL / Data Processing"
        ETL1["prep/ Python & Notebooks"]:::compute
        ETL2["impacts/ Jupyter & Python"]:::compute
        ETL3["trade/ scripts"]:::compute
        ETL4["crm/ pipelines"]:::compute
        ETL5["salesforce update scripts"]:::compute
        ETL6["databricks notebooks"]:::compute
    end

    subgraph "Data Storage"
        DS1["Flat Files (CSV/JSON/Parquet)"]:::storage
        DS2["DuckDB Databases"]:::storage
        DS3["Supabase Postgres"]:::storage
    end

    subgraph "Static Site Generator / Build"
        BUILD1["azure/ deployment script"]:::compute
        BUILD2["React/Vite build"]:::compute
    end

    subgraph "Frontend Presentation"
        FE1["charts micro-frontends"]:::frontend
        FE2["food-nutrition-app"]:::frontend
    end

    subgraph "Hosting / CDN"
        HOST["Azure Static Web Apps & CDN"]:::hosting
    end

    subgraph "End User Browser"
        BROWSER["Browser"]:::external
    end

    EDS1 --> ETL1
    EDS1 --> ETL2
    EDS2 --> ETL1
    EDS3 --> ETL3
    EDS4 --> ETL5

    ETL1 --> DS1
    ETL1 --> DS2
    ETL1 --> DS3
    ETL2 --> DS1
    ETL2 --> DS2
    ETL2 --> DS3
    ETL3 --> DS1
    ETL3 --> DS2
    ETL3 --> DS3
    ETL4 --> DS1
    ETL4 --> DS2
    ETL4 --> DS3
    ETL5 --> DS1
    ETL5 --> DS2
    ETL5 --> DS3
    ETL6 --> DS1
    ETL6 --> DS2
    ETL6 --> DS3

    DS1 --> BUILD1
    DS1 --> BUILD2
    DS2 --> BUILD1
    DS2 --> BUILD2
    DS3 --> BUILD1
    DS3 --> BUILD2

    BUILD1 --> HOST
    BUILD2 --> HOST

    HOST --> BROWSER

    BROWSER -.->|"runtime query"|DS3
    BROWSER -.->|"runtime DuckDB"|DS2

    click EDS1 "https://github.com/modelearth/profile/tree/main/impacts/exiobase/US-source/"
    click EDS2 "https://github.com/modelearth/profile/blob/main/prep/naics/2022-NAICS-Codes-listed-numerically-2-Digit-through-6-Digit.xlsx"
    click EDS3 "https://github.com/modelearth/profile/tree/main/trade/naics/"
    click EDS4 "https://github.com/modelearth/profile/blob/main/salesforce/directory/update/csv.py"

    click ETL1 "https://github.com/modelearth/profile/tree/main/prep/"
    click ETL2 "https://github.com/modelearth/profile/tree/main/impacts/exiobase/US-source/"
    click ETL3 "https://github.com/modelearth/profile/tree/main/trade/"
    click ETL4 "https://github.com/modelearth/profile/tree/main/crm/"
    click ETL5 "https://github.com/modelearth/profile/blob/main/salesforce/directory/update/csv.py"
    click ETL6 "https://github.com/modelearth/profile/tree/main/databricks/"

    click DS1 "https://github.com/modelearth/profile/tree/main/impacts/json/"
    click DS2 "https://github.com/modelearth/profile/blob/main/impacts/exiobase/US-source/US-2020-17schema.duckdb"
    click DS3 "https://github.com/modelearth/profile/blob/main/prep/sql/supabase/database_backup.sql"

    click BUILD1 "https://github.com/modelearth/profile/blob/main/azure/azure.sh"
    click BUILD2 "https://github.com/modelearth/profile/blob/main/item/food-nutrition-app/vite.config.js"

    click FE1 "https://github.com/modelearth/profile/tree/main/charts/"
    click FE2 "https://github.com/modelearth/profile/tree/main/item/food-nutrition-app/src/"

    click HOST "https://github.com/modelearth/profile/blob/main/azure/create-database2_site.yml"

    classDef compute fill:#ADD8E6
    classDef storage fill:#90EE90
    classDef external fill:#D3D3D3
    classDef frontend fill:#FFA500
    classDef hosting fill:#D8BFD8
```
