How mohamed-12-tarek/egypt-flash-flood-early-warning works
This repository is organized as an Egypt flash-flood early-warning system: data acquisition and streaming feed data processing and storage, which support risk inference and user-facing analysis/chat. The tree identifies these responsibilities, but most sampled implementation files are empty; the relationships below therefore reflect only the flow indicated by component names, not verified call details.
Main entry points: App.jsx Web interface · historical_bootstrap_job.py Historical bootstrap · live_incremental_job.py Live processing · analysis_api.py Analysis API
Components 29
Data acquisition
- Historical fetch
ingestion/fetch_historical.py - Static data fetch
ingestion/fetch_static_data.py - Live producer
ingestion/producer_live.py - Bronze consumer
ingestion/kafka_to_bronze_consumer.py
Processing and storage
- Historical bootstrap
spark/historical_bootstrap_job.py - Live processing
spark/live_incremental_job.py - Shared features
spark/shared_feature_logic.py - Silver-to-gold load
spark/silver_to_gold_load.py - Silver schema
warehouse/sql/create_silver_table.sql - Gold schema
warehouse/sql/create_gold_tables.sql - SQL Server
- DB connection
warehouse/db_connection.py
Risk and analysis
- Model training
ml/train_model.py - Model retraining
ml/retrain_job.py - Risk inference
ml/inference_job.py - Analysis API
analysis/analysis_api.py - Analysis queries
analysis/analysis_queries.py - KPI calculations
analysis/kpi_calculations.py
User experience
- Web interface
frontend/src/App.jsx - Analysis panel
frontend/src/components/AnalysisPanel.jsx - Alert banner
frontend/src/components/AlertBanner.jsx - Chat panel
frontend/src/components/ChatbotPanel.jsx - Chatbot API
genai/chatbot_api.py - Chatbot chain
genai/chatbot_chain.py - Data to text
genai/data_to_text.py - Vector store setup
genai/vector_store_setup.py
Other components
- Data sources
- Kafka
- User
Connections 30
- Data sources → Historical fetch: fetches history
- Data sources → Static data fetch: fetches static data
- Data sources → Live producer: streams updates
- Live producer → Kafka: publishes
- Kafka → Bronze consumer: delivers messages
- Historical fetch → Historical bootstrap: bootstraps
- Static data fetch → Historical bootstrap: bootstraps
- Bronze consumer → Live processing: provides bronze data
- Historical bootstrap → Shared features: uses features
- Live processing → Shared features: uses features
- Historical bootstrap → Silver schema: loads silver data
- Live processing → Silver schema: loads silver data
- Silver schema → SQL Server: defines table
- Gold schema → SQL Server: defines tables
- Silver-to-gold load → SQL Server: loads gold data
- DB connection → SQL Server: connects to
- Shared features → Model training: provides features
- Model training → Risk inference: provides model
- Model retraining → Risk inference: provides model
- User → Web interface: uses
- Web interface → Analysis panel: displays
- Web interface → Alert banner: displays
- Web interface → Chat panel: displays
- Analysis panel → Analysis API: requests analysis
- Analysis API → Analysis queries: uses queries
- Analysis API → KPI calculations: calculates KPIs
- Chat panel → Chatbot API: sends questions
- Chatbot API → Chatbot chain: dispatches
- Chatbot chain → Data to text: formats data
- Chatbot chain → Vector store setup: uses retrieval