How modelearth/data-pipeline works
Project type and purpose
• This is a hybrid “JAMstack + data‐pipeline” repository.
• A static site (root HTML, industries/, timelines/, research/, us/, etc.) for documentation, reports, and data views.
• A Next.js‐based admin interface under admin/ (React components + serverless API routes under pages/api).
• A collection of Python scripts, Jupyter notebooks, and CSV/JSON files implementing the actual ETL/data‐prep pipelines.
• CI/CD (GitHub Actions) to build and deploy both the static site and admin UI.
File structure
• Top‐level:
• .github/workflows/deploy.yml + DEPLOYMENT.md → CI/CD pipeline.
• admin/ → Next.js app (React + Tailwind) with serverless API:
– components/, pages/, pages/api/nodes/run.js, nodes/update.js → CRUD on node definitions.
– utils/csvParser.js → CSV ↔ JSON conversion.
• Scripts & data directories (no centralized “backend” service):
– timelines/prep/ (Python ETL scripts reading raw CSV, generating outputs).
– research/, industries/, international/, us/ → data sources, notebooks, CSVs.
• Static site output under admin/out/ and admin/.next/ for export.
• n8n_workflow_sample.json, nodes.csv, nodes.json → pipeline definitions.
README for additional insights
• Purpose: “Data Pipeline” linking to community data, timeliness, IO charts, international trade, USEEIO, Data Commons.
Detail is key
• API endpoint (run, update) as its own function.
• File paths or directory names where critical data lives.
• ETL stages (input folder → script → output folder).
• Make explicit any triggers (e.g., Admin “Run Pipeline” button → serverless function → Python ETL job).
• Don’t hesitate to break the data pipelines into multiple components if it clarifies flow.