How eparirishit/clean-and-green-philly works
A client‐facing web application built with React, NextJS, and Tailwind CSS.
2. A Python-powered data engineering pipeline (including ETL processes) that handles data ingestion, transformation, and possibly some computational steps (with some C++ integrations in the “awkde” module) to support the dashboard’s functionality.
The overall purpose is to empower community groups by providing a dashboard that helps identify vacant properties where cleaning and greening interventions can reduce gun violence in Philadelphia. The system relies on both a modern web frontend for data visualization and user interaction, as well as backend data pipelines for ETL and processing of public datasets that drive decision making.
Main Components & Their Roles
• Frontend (Client Web App):
– Built using React and NextJS, organized in the “src/app” folder.
– Contains pages, components (map, property details, informational content), styling (Tailwind CSS), and utilities.
– Acts as the primary interface for users, handling navigation, interactive maps, and property information.
• Backend Data Pipeline:
– Located mainly in the “data/src” directory. This module includes ETL code, data utilities, configuration files, and tests.
– Contains submodules such as “new_etl” and “awkde” (which even integrates C++ via “cpp/backend.cpp”) to process and enhance the data.
– Prepares datasets, runs data transformations, and may update the underlying database tables.
• Database:
– Dockerfiles (Dockerfile, Dockerfile-pg, Dockerfile-timescale) and SQL scripts (init_pg.sql, init_pg_timescale.sql) indicate that PostgreSQL (and possibly TimescaleDB for time-series optimizations) are used.
– Acts as persistent storage for the cleaned, processed data that the frontend displays.
Infrastructure and DevOps Components
• Docker & Containerization:
– The “data” folder contains Dockerfiles and a “docker-compose.yml” that likely orchestrates running the database containers and associated back-end services.
• Testing and CI/CD Pipelines:
– The presence of a “.github” folder with configuration for GitHub Actions and additional workflows (e.g., pr_checks, release workflows) indicates automated testing, linting, and release management processes.
• Configuration & Build Tools:
– ESLint and Prettier configuration, TypeScript configurations (tsconfig.json), and front-end build configurations (next.config.js) illustrate that the project is set up with best practices in code quality and build processes.
Relationships and Data Flow
• The Frontend sends user interactions (map requests, property queries) likely via API calls (either directly or through server-side fetching in NextJS) which retrieve data from the backend.
• The Python ETL pipeline (running as scheduled jobs or manual triggers) consumes raw public data, processes it (potentially using both Python modules and C++ acceleration in “awkde”), and then updates the PostgreSQL/Timescale databases.
• The database then serves as a data source for the web application, perhaps via API routes (either embedded in NextJS or as a separate API service).
• DevOps pipelines (CI/CD through GitHub Actions) ensure tests and build steps are executed, which then drive deployments for both the React app and the Python backend.
Architectural Design Principles
• Separation of Concerns: Clear split between frontend (display and interaction) and backend (data processing and storage).
• Containerization: Use of Docker and docker-compose helps isolate the database and possibly the ETL process.
• Modular Design: Within the Python pipeline, directories are neatly organized into “data_utils”, “classes”, “new_etl”, etc., emphasizing modular, reusable components.
• Automated Testing & CI: Testing directories (e.g., “data/src/test”) and GitHub workflows ensure that quality is enforced throughout the development lifecycle.