How chaurasiavikash/super_emitter_tracking works
Project Type and Purpose
• This is a Python-based, full-stack data processing and monitoring system for detecting, tracking, and analyzing methane “super-emitters” from satellite (TROPOMI) data.
• It combines an ETL pipeline (data collection → preprocessing → detection → tracking), an alert/notification subsystem, and an interactive Streamlit dashboard for real-time monitoring and historical analysis.
File Structure Analysis
• src/
• data/ → ingestion (TROPOMI collector, meteorology, reference data) + validation/preprocessing
• detection/ → core detection algorithms (change detector, plume analysis, ensemble voting)
• tracking/ → temporal association and trend analysis (persistence filter, trend detector)
• alerts/ → alert manager, threshold monitoring, notification sender
• visualization/ → Streamlit dashboard components, map & time-series plotting, report generator
• analysis/ → statistical analyzers, uncertainty estimators, seasonal decomposers
• validation/ → cross-validation, ground truth comparison, performance metrics
• utils/ → shared helpers (file IO, geospatial, time, statistics)
• main.py → CLI/orchestrator driving end-to-end pipeline
• config/ → YAML and Python config for thresholds, DB connection, logging
• deployment/ → Dockerfile, docker-compose, Kubernetes YAML, API server and Streamlit app entrypoints
• scripts/ → batch jobs (database setup, report generation, validation runs)
• notebooks/ → ad-hoc exploratory and case-study notebooks
• tests/ → pytest suite for each module
README Insights
• Processing modes: test, operational (real-time), historical
• External dependencies: Google Earth Engine, SQL database (emitter_database), email/webhook endpoints
• Key algorithms outlined: ensemble detection, DBSCAN clustering, Mann–Kendall trend tests
• System architecture summary: Data Collection → Preprocessing → Detection → Tracking → Alerts
• Data Ingestion Layer (src/data/*)
• Preprocessing & Validation (data_validator, preprocessor)
• Detection Engine (src/detection/* + analysis modules)
• Tracking Engine (src/tracking/*)
• Alerting Subsystem (src/alerts/*)
• Visualization Layer (src/visualization/* & Streamlit app)
• Storage Layer
• Configuration store (config/config.yaml)
• Time-series & detection records (emitter_database)
• Flat files/exports (CSV, GeoJSON, NetCDF)
• External Services
• Google Earth Engine (satellite data API)
• Notification endpoints (SMTP, webhooks)
• Orchestrator/API Server (src/main.py, deployment/api_server.py)
1. External Data Services → Data Ingestion
2. Ingestion → Validation → Preprocessing
3. Preprocessed data → Detection Engine
4. Detections → Tracking Engine → Trend & change analysis
5. Tracking outputs →
• Storage Layer
• Alerting Subsystem → Notification endpoints
• Visualization Dashboard → End user
6. Configuration store feeds thresholds & DB credentials to all modules
7. Orchestrator/API coordinates batch/real-time runs and serves dashboard via Streamlit
c. Architectural Patterns & Design Principles
• Layered Pipeline: clear separation of data, processing, and presentation
• Modularity: each algorithm lives in its own package (opens room for plugin-style extension)
• Config-driven: all tuning parameters in YAML; no hard-coding
• Microservice-style deployment: API server + dashboard + batch runner in separate containers
• Reproducibility: scripts, notebooks, and tests ensure consistency
d. Key Technologies & Frameworks
• Python 3.9+ with packages defined in requirements.txt
• Google Earth Engine Python API
• Pandas, NumPy, SciPy, scikit-learn (DBSCAN)
• Streamlit for dashboard
• SQL database (configured via emitter_database.yaml)
• Docker, Docker-Compose, Kubernetes