How hossamtabana/terrametria works
Project type and purpose
• Terrametria is a full-stack, data-driven web application that renders an interactive 3D population density map in the browser.
• Core responsibilities:
• Data ingestion & processing (Databricks workflows)
• Data storage & serving (Databricks SQL Warehouse)
• Backend API (FastAPI as a Databricks App)
• Frontend SPA (React + Deck.gl)
File structure for architectural clues
• Top-level folders:
• src/frontend → React/Vite single-page application
• src/terrametria → Python backend modules (entrypoint, loader, API)
• databricks.yml, hooks/yarn_build.py → CI/CD & Databricks bundle config
• Tests target the API layer (tests/test_api.py).
• Configuration files (.env.example, pyproject.toml, package.json, vite.config.ts) hint at two distinct build systems.
README for additional context
• Workflow:
1. Databricks bundle deploy & run → executes Python loader + workflows.
2. Data lands in a SQL Warehouse (catalog + schema).
3. Backend and frontend run locally (hatch dev-backend / yarn dev-frontend) or deployed as a Databricks App.
• Tech stack summary (React, TypeScript, FastAPI, Databricks Apps, Databricks SQL).
a. Main components
• External Data Source (HDX/Meta)
• Databricks Compute (Bundles & Jobs)
• Databricks SQL Warehouse (persistent storage)
• Backend Service (FastAPI running as a Databricks App)
• Frontend App (React + Deck.gl on Vite)
• Client Browser
b. Relationships & data-flow
1. Loader module in Databricks Compute fetches HDX data → transforms → writes to SQL tables.
2. Backend Service connects to SQL Warehouse using Client ID/Secret → exposes HTTP endpoints (API).
3. Frontend App makes HTTP requests to Backend (GET density‐map data) → processes JSON → displays via Deck.gl.
4. Developer workflows:
• Local: hatch run dev-backend ↔ local FastAPI
• yarn dev-frontend ↔ local Vite server
• Deployed: Databricks App hosts both backend and frontend (static assets).
c. Important patterns & principles
• Clear separation of concerns: Data ingestion, storage, API layer, UI layer
• Environment-driven configuration (DATABRICKS_HOST, CLIENT_ID/SECRET, CATALOG, SCHEMA)
• Infrastructure as code via databricks.yml and bundle commands
• React componentization (components/ui, pages, lib)
d. Key technologies to annotate
• Deck.gl layer in the browser
• FastAPI endpoints (e.g., /density)
• Databricks CLI & Apps lifecycle
• SQL Warehouse (catalog.schema.table)