How shahilcseai/gdg-hackathon works
Project Type and Purpose
• This is a Python-based data-processing and ML pipeline whose goal is to downscale coarse-resolution satellite NO₂ data to a fine-resolution air-quality map.
• It ingests multiple data sources (satellite swath/gridded TIFF, ground station CSV), applies pre‐processing, trains or applies an ML model, validates results, and produces final maps.
File Structure Analysis
• Top-level scripts:
– main.py: orchestrates end-to-end workflow (ingest → preprocess → train/evaluate → output)
– database.py: handles reading/writing intermediate data (e.g., local cache or PostGIS)
– model.py: defines/selects/trains the ML algorithm (RandomForest, XGBoost, CNN, etc.)
– utils.py: shared utilities (e.g., data cleaning, gap filling for cloudy pixels, coordinate transforms)
• styles.css: styling for any HTML/JS map visualizations or reports
• pyproject.toml: project metadata and dependencies
• README.md: high-level description, data sources, algorithm choices, usage
A. External Data Sources
• Satellite NO₂ sources (TROPOMI/Sentinel-5p swath or gridded; OMI Aura)
• Ground-station CPCB CSV/API
B. Ingestion Layer
• Scripts in database.py and utils.py that fetch, cache, and standardize raw data
C. Pre-processing Layer
• Gap‐filling logic for cloudy pixels
• Spatial re-projection and gridding
• Merging satellite + ground data
D. ML Model Layer
• model.py: training pipeline or inference pipeline
• Hyperparameter tuning (RF, XGBoost, ANN/CNN)
E. Validation Layer
• Hold-out data evaluation (unseen independent)
• Accuracy metrics (RMSE, spatial error maps)
F. Output/Visualization Layer
• Fine-resolution NO₂ maps (GeoTIFF, shapefile)
• HTML dashboard or report styled via styles.css
G. Orchestration
• main.py coordinates:
– calls ingestion → pre‐processing → model training/inference → validation → reporting
Relationships and Data Flows
1. External Data Sources → Ingestion Layer
2. Ingestion → Pre-processing
3. Pre-processing → ML Model Layer
4. ML Model Layer ↔ Validation Layer (bidirectional during training/tuning)
5. ML Model Layer → Output/Visualization
6. main.py (orchestrator) invokes each component in sequence
• That database.py and utils.py are shared libraries supporting multiple stages.
Architectural Patterns and Design Principles
• Modular pipeline: separate ingestion, preprocessing, modeling, validation, and reporting
• Separation of concerns:
• database.py handles I/O, utils.py handles generic transformations, model.py focuses on ML logic
• Configurable ML back-end: plug-and-play algorithms (RF, XGBoost, CNN)
• Stateless steps: each stage reads inputs, writes outputs—can parallelize or run on different machines
Technologies, Frameworks, and External Services
• Python ML libraries: scikit-learn, XGBoost, TensorFlow/PyTorch
• Geospatial libraries: rasterio, geopandas, shapely
• Data ingestion: requests, earthengine-api (for GEE)
• Reporting: folium or plotly for interactive maps, HTML/CSS (styles.css)
• (Optional) Database: SQLite/PostGIS via SQLAlchemy or custom code in database.py