# modelearth/data-pipeline architecture

> Architecture diagram of the GitHub repository modelearth/data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/modelearth/data-pipeline
- Repository: https://github.com/modelearth/data-pipeline

## Overview

##### Project type and purpose
- This is a hybrid “JAMstack + data‐pipeline” repository.
- A static site (root HTML, `industries/`, `timelines/`, `research/`, `us/`, etc.) for documentation, reports, and data views.
- A Next.js‐based admin interface under `admin/` (React components + serverless API routes under `pages/api`).
- A collection of Python scripts, Jupyter notebooks, and CSV/JSON files implementing the actual ETL/data‐prep pipelines.
- CI/CD (GitHub Actions) to build and deploy both the static site and admin UI.

##### File structure
- Top‐level:
- `.github/workflows/deploy.yml` + `DEPLOYMENT.md` → CI/CD pipeline.
- `admin/` → Next.js app (React + Tailwind) with serverless API:
– `components/`, `pages/`, `pages/api/nodes/run.js`, `nodes/update.js` → CRUD on node definitions.
– `utils/csvParser.js` → CSV ↔ JSON conversion.
- Scripts & data directories (no centralized “backend” service):
– `timelines/prep/` (Python ETL scripts reading raw CSV, generating outputs).
– `research/`, `industries/`, `international/`, `us/` → data sources, notebooks, CSVs.
- Static site output under `admin/out/` and `admin/.next/` for export.
- `n8n_workflow_sample.json`, `nodes.csv`, `nodes.json` → pipeline definitions.

##### README for additional insights
- Purpose: “Data Pipeline” linking to community data, timeliness, IO charts, international trade, USEEIO, Data Commons.

##### Detail is key
- API endpoint (`run`, `update`) as its own function.
- File paths or directory names where critical data lives.
- ETL stages (input folder → script → output folder).
- Make explicit any triggers (e.g., Admin “Run Pipeline” button → serverless function → Python ETL job).
- Don’t hesitate to break the data pipelines into multiple components if it clarifies flow.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Developer & CI/CD
    subgraph "Developer & CI/CD"
        Dev["Developer"] 
        Actions["GitHub Actions"]:::external
        YML["CI/CD Config"]:::api
        MD["Deployment Docs"]:::data
    end
    Dev --> Actions
    Actions --> YML
    Actions --> MD
    click YML "https://github.com/modelearth/data-pipeline/blob/main/.github/workflows/deploy.yml"
    click MD "https://github.com/modelearth/data-pipeline/blob/main/DEPLOYMENT.md"

    %% Static Site Content
    subgraph "Static Site Content"
        RootHTML["index.html"]:::data
        Ind["industries/"]:::data
        TimDoc["timelines/"]:::data
        Res["research/"]:::data
        US["us/"]:::data
        Intl["international/"]:::data
    end
    click RootHTML "https://github.com/modelearth/data-pipeline/blob/main/index.html"
    click Ind "https://github.com/modelearth/data-pipeline/tree/main/industries/"
    click TimDoc "https://github.com/modelearth/data-pipeline/tree/main/timelines/"
    click Res "https://github.com/modelearth/data-pipeline/tree/main/research/"
    click US "https://github.com/modelearth/data-pipeline/tree/main/us/"
    click Intl "https://github.com/modelearth/data-pipeline/tree/main/international/"

    %% Static Site Build & Hosting
    subgraph "Static Site Build & Hosting"
        SSBuild["Build Static Site"]:::api
        SSHost["CDN / GitHub Pages"]:::external
    end
    Actions --> SSBuild
    SSBuild --> SSHost
    RootHTML & Ind & TimDoc & Res & US & Intl --> SSBuild

    %% Admin UI & API Layer
    subgraph "Admin UI & API"
        Browser["Admin User Browser"]:::frontend
        AdminUI["Next.js Admin UI"]:::frontend
        Components["React Components"]:::frontend
        Pages["Pages (Frontend Routes)"]:::frontend
        Styles["Global Styles"]:::data
        TailConf["Tailwind Config"]:::data
        NextConf["Next.js Config"]:::data
        CSVParser["csvParser.js"]:::api
        API["API Layer"]:::api
        NodesJS["nodes.js"]:::api
        RunFn["/api/nodes/run"]:::api
        UpdateFn["/api/nodes/update"]:::api
    end
    click AdminUI "https://github.com/modelearth/data-pipeline/tree/main/admin/"
    click Components "https://github.com/modelearth/data-pipeline/tree/main/admin/components/"
    click Pages "https://github.com/modelearth/data-pipeline/tree/main/admin/pages/"
    click Styles "https://github.com/modelearth/data-pipeline/blob/main/admin/styles/globals.css"
    click TailConf "https://github.com/modelearth/data-pipeline/blob/main/admin/tailwind.config.js"
    click NextConf "https://github.com/modelearth/data-pipeline/blob/main/admin/next.config.js"
    click CSVParser "https://github.com/modelearth/data-pipeline/blob/main/admin/utils/csvParser.js"
    click NodesJS "https://github.com/modelearth/data-pipeline/blob/main/admin/pages/api/nodes.js"
    click RunFn "https://github.com/modelearth/data-pipeline/blob/main/admin/pages/api/nodes/run.js"
    click UpdateFn "https://github.com/modelearth/data-pipeline/blob/main/admin/pages/api/nodes/update.js"

    Browser --> AdminUI
    AdminUI --> Components
    AdminUI --> Pages
    AdminUI --> Styles
    AdminUI --> TailConf
    AdminUI --> NextConf
    AdminUI --> CSVParser
    AdminUI --> API
    API --> NodesJS
    NodesJS --> RunFn
    NodesJS --> UpdateFn
    Browser -->|invokes| RunFn
    Browser -->|invokes| UpdateFn

    %% Data Store
    subgraph "Flat-File Data Store"
        CSV["nodes.csv"]:::data
        JSON["nodes.json"]:::data
        N8N["n8n_workflow_sample.json"]:::data
    end
    click CSV "https://github.com/modelearth/data-pipeline/blob/main/nodes.csv"
    click JSON "https://github.com/modelearth/data-pipeline/blob/main/nodes.json"
    click N8N "https://github.com/modelearth/data-pipeline/blob/main/n8n_workflow_sample.json"
    RunFn --> CSV
    RunFn --> JSON
    UpdateFn --> CSV
    UpdateFn --> JSON

    %% ETL Batch Jobs
    subgraph "ETL Batch Jobs"
        AllPrep["Timelines All Pipeline"]:::etl
        RegPrep["Timelines Regression Pipeline"]:::etl
        IndPrep["Timelines Industries Pipeline"]:::etl
        PovertyNB["Poverty Prep Notebooks"]:::etl
        NAICS["Industries NAICS Scripts"]:::etl
        Comtrade["International COMTRADE Scripts"]:::etl
        Zip["US Zipcode Scripts"]:::etl
        CountiesStates["US Counties & States Docs"]:::etl
        AllOutput["All Pipeline Output"]:::data
    end
    click AllPrep "https://github.com/modelearth/data-pipeline/tree/main/timelines/prep/all/"
    click RegPrep "https://github.com/modelearth/data-pipeline/tree/main/timelines/prep/regression/"
    click IndPrep "https://github.com/modelearth/data-pipeline/tree/main/timelines/prep/industries/"
    click PovertyNB "https://github.com/modelearth/data-pipeline/tree/main/timelines/prep/all/input/poverty/python/"
    click NAICS "https://github.com/modelearth/data-pipeline/tree/main/industries/naics/"
    click Comtrade "https://github.com/modelearth/data-pipeline/tree/main/international/comtrade/"
    click Zip "https://github.com/modelearth/data-pipeline/tree/main/us/zipcodes/"
    click CountiesStates "https://github.com/modelearth/data-pipeline/tree/main/us/counties/"
    AllPrep --> AllOutput
    RegPrep --> AllOutput
    IndPrep --> AllOutput
    Ind & TimDoc & Res & US & Intl & CSV & JSON --> AllPrep
    RegPrep --> AllOutput
    IndPrep --> AllOutput
    PovertyNB --> AllPrep
    NAICS --> IndPrep
    Comtrade --> IndPrep
    Zip --> RegPrep
    CountiesStates --> RegPrep
    RunFn --> AllPrep

    %% External Services
    subgraph "External Services"
        GDC["Google Data Commons API"]:::external
        USEEIO["USEEIO API"]:::external
    end
    AllPrep --> GDC
    AllPrep --> USEEIO
    RegPrep --> GDC
    RegPrep --> USEEIO
    IndPrep --> GDC

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef api fill:#90EE90,stroke:#333,stroke-width:1px
    classDef data fill:#FFFFE0,stroke:#333,stroke-width:1px
    classDef etl fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
```
