# mohamed-12-tarek/egypt-flash-flood-early-warning architecture

> Architecture diagram of the GitHub repository mohamed-12-tarek/egypt-flash-flood-early-warning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohamed-12-tarek/egypt-flash-flood-early-warning
- Repository: https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning

## Overview

This repository is organized as an Egypt flash-flood early-warning system: data acquisition and streaming feed data processing and storage, which support risk inference and user-facing analysis/chat. The tree identifies these responsibilities, but most sampled implementation files are empty; the relationships below therefore reflect only the flow indicated by component names, not verified call details.

Main entry points: [`frontend/src/App.jsx`](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/App.jsx) (Web interface), [`spark/historical_bootstrap_job.py`](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/historical_bootstrap_job.py) (Historical bootstrap), [`spark/live_incremental_job.py`](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/live_incremental_job.py) (Live processing), [`analysis/analysis_api.py`](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/analysis/analysis_api.py) (Analysis API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Data acquisition"]
  node_historical_fetch["Historical fetch"]
  node_static_fetch["Static data fetch"]
  node_live_producer["Live producer<br/>[producer_live.py]"]
  node_bronze_consumer["Bronze consumer"]
end

subgraph group_data["Processing and storage"]
  node_historical_spark["Historical bootstrap"]
  node_live_spark["Live processing"]
  node_feature_logic["Shared features"]
  node_silver_gold["Silver-to-gold load"]
  node_silver_schema["Silver schema"]
  node_gold_schema["Gold schema"]
  node_sql_server[("SQL Server")]
  node_db_connection["DB connection<br/>[db_connection.py]"]
end

subgraph group_forecast["Risk and analysis"]
  node_train_model["Model training<br/>[train_model.py]"]
  node_retrain_model["Model retraining<br/>[retrain_job.py]"]
  node_inference["Risk inference<br/>[inference_job.py]"]
  node_analysis_api["Analysis API<br/>[analysis_api.py]"]
  node_analysis_queries["Analysis queries"]
  node_kpi["KPI calculations"]
end

subgraph group_experience["User experience"]
  node_frontend["Web interface<br/>[App.jsx]"]
  node_analysis_panel["Analysis panel<br/>[AnalysisPanel.jsx]"]
  node_alert_banner["Alert banner<br/>[AlertBanner.jsx]"]
  node_chat_panel["Chat panel<br/>[ChatbotPanel.jsx]"]
  node_chat_api["Chatbot API<br/>[chatbot_api.py]"]
  node_chat_chain["Chatbot chain<br/>[chatbot_chain.py]"]
  node_data_to_text["Data to text<br/>[data_to_text.py]"]
  node_vector_store["Vector store setup"]
end

node_data_sources(("Data sources"))
node_kafka["Kafka"]
node_user(("User"))

node_data_sources -->|"fetches history"| node_historical_fetch
node_data_sources -->|"fetches static data"| node_static_fetch
node_data_sources -->|"streams updates"| node_live_producer
node_live_producer -->|"publishes"| node_kafka
node_kafka -->|"delivers messages"| node_bronze_consumer
node_historical_fetch -->|"bootstraps"| node_historical_spark
node_static_fetch -->|"bootstraps"| node_historical_spark
node_bronze_consumer -->|"provides bronze data"| node_live_spark
node_historical_spark -->|"uses features"| node_feature_logic
node_live_spark -->|"uses features"| node_feature_logic
node_historical_spark -->|"loads silver data"| node_silver_schema
node_live_spark -->|"loads silver data"| node_silver_schema
node_silver_schema -->|"defines table"| node_sql_server
node_gold_schema -->|"defines tables"| node_sql_server
node_silver_gold -->|"loads gold data"| node_sql_server
node_db_connection -->|"connects to"| node_sql_server
node_feature_logic -->|"provides features"| node_train_model
node_train_model -->|"provides model"| node_inference
node_retrain_model -->|"provides model"| node_inference
node_user -->|"uses"| node_frontend
node_frontend -->|"displays"| node_analysis_panel
node_frontend -->|"displays"| node_alert_banner
node_frontend -->|"displays"| node_chat_panel
node_analysis_panel -->|"requests analysis"| node_analysis_api
node_analysis_api -->|"uses queries"| node_analysis_queries
node_analysis_api -->|"calculates KPIs"| node_kpi
node_chat_panel -->|"sends questions"| node_chat_api
node_chat_api -->|"dispatches"| node_chat_chain
node_chat_chain -->|"formats data"| node_data_to_text
node_chat_chain -->|"uses retrieval"| node_vector_store

click node_historical_fetch "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ingestion/fetch_historical.py"
click node_static_fetch "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ingestion/fetch_static_data.py"
click node_live_producer "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ingestion/producer_live.py"
click node_bronze_consumer "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ingestion/kafka_to_bronze_consumer.py"
click node_historical_spark "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/historical_bootstrap_job.py"
click node_live_spark "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/live_incremental_job.py"
click node_feature_logic "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/shared_feature_logic.py"
click node_silver_gold "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/silver_to_gold_load.py"
click node_silver_schema "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/warehouse/sql/create_silver_table.sql"
click node_gold_schema "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/warehouse/sql/create_gold_tables.sql"
click node_db_connection "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/warehouse/db_connection.py"
click node_train_model "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ml/train_model.py"
click node_retrain_model "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ml/retrain_job.py"
click node_inference "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ml/inference_job.py"
click node_analysis_api "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/analysis/analysis_api.py"
click node_analysis_queries "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/analysis/analysis_queries.py"
click node_kpi "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/analysis/kpi_calculations.py"
click node_frontend "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/App.jsx"
click node_analysis_panel "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/components/AnalysisPanel.jsx"
click node_alert_banner "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/components/AlertBanner.jsx"
click node_chat_panel "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/components/ChatbotPanel.jsx"
click node_chat_api "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/genai/chatbot_api.py"
click node_chat_chain "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/genai/chatbot_chain.py"
click node_data_to_text "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/genai/data_to_text.py"
click node_vector_store "https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/genai/vector_store_setup.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_historical_fetch,node_static_fetch,node_live_producer,node_bronze_consumer,node_user toneBlue
class node_historical_spark,node_live_spark,node_feature_logic,node_silver_gold,node_silver_schema,node_gold_schema,node_sql_server,node_db_connection toneAmber
class node_train_model,node_retrain_model,node_inference,node_analysis_api,node_analysis_queries,node_kpi toneMint
class node_frontend,node_analysis_panel,node_alert_banner,node_chat_panel,node_chat_api,node_chat_chain,node_data_to_text,node_vector_store toneRose
class node_data_sources,node_kafka toneIndigo
```

## Components

### Data acquisition

- [Historical fetch](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ingestion/fetch_historical.py): `ingestion/fetch_historical.py`
- [Static data fetch](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ingestion/fetch_static_data.py): `ingestion/fetch_static_data.py`
- [Live producer](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ingestion/producer_live.py): `ingestion/producer_live.py`
- [Bronze consumer](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ingestion/kafka_to_bronze_consumer.py): `ingestion/kafka_to_bronze_consumer.py`

### Processing and storage

- [Historical bootstrap](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/historical_bootstrap_job.py): `spark/historical_bootstrap_job.py`
- [Live processing](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/live_incremental_job.py): `spark/live_incremental_job.py`
- [Shared features](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/shared_feature_logic.py): `spark/shared_feature_logic.py`
- [Silver-to-gold load](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/spark/silver_to_gold_load.py): `spark/silver_to_gold_load.py`
- [Silver schema](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/warehouse/sql/create_silver_table.sql): `warehouse/sql/create_silver_table.sql`
- [Gold schema](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/warehouse/sql/create_gold_tables.sql): `warehouse/sql/create_gold_tables.sql`
- **SQL Server**
- [DB connection](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/warehouse/db_connection.py): `warehouse/db_connection.py`

### Risk and analysis

- [Model training](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ml/train_model.py): `ml/train_model.py`
- [Model retraining](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ml/retrain_job.py): `ml/retrain_job.py`
- [Risk inference](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/ml/inference_job.py): `ml/inference_job.py`
- [Analysis API](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/analysis/analysis_api.py): `analysis/analysis_api.py`
- [Analysis queries](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/analysis/analysis_queries.py): `analysis/analysis_queries.py`
- [KPI calculations](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/analysis/kpi_calculations.py): `analysis/kpi_calculations.py`

### User experience

- [Web interface](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/App.jsx): `frontend/src/App.jsx`
- [Analysis panel](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/components/AnalysisPanel.jsx): `frontend/src/components/AnalysisPanel.jsx`
- [Alert banner](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/components/AlertBanner.jsx): `frontend/src/components/AlertBanner.jsx`
- [Chat panel](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/frontend/src/components/ChatbotPanel.jsx): `frontend/src/components/ChatbotPanel.jsx`
- [Chatbot API](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/genai/chatbot_api.py): `genai/chatbot_api.py`
- [Chatbot chain](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/genai/chatbot_chain.py): `genai/chatbot_chain.py`
- [Data to text](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/genai/data_to_text.py): `genai/data_to_text.py`
- [Vector store setup](https://github.com/mohamed-12-tarek/egypt-flash-flood-early-warning/blob/main/genai/vector_store_setup.py): `genai/vector_store_setup.py`

### Other components

- **Data sources**
- **Kafka**
- **User**

## Connections

- Data sources → Historical fetch: fetches history
- Data sources → Static data fetch: fetches static data
- Data sources → Live producer: streams updates
- Live producer → Kafka: publishes
- Kafka → Bronze consumer: delivers messages
- Historical fetch → Historical bootstrap: bootstraps
- Static data fetch → Historical bootstrap: bootstraps
- Bronze consumer → Live processing: provides bronze data
- Historical bootstrap → Shared features: uses features
- Live processing → Shared features: uses features
- Historical bootstrap → Silver schema: loads silver data
- Live processing → Silver schema: loads silver data
- Silver schema → SQL Server: defines table
- Gold schema → SQL Server: defines tables
- Silver-to-gold load → SQL Server: loads gold data
- DB connection → SQL Server: connects to
- Shared features → Model training: provides features
- Model training → Risk inference: provides model
- Model retraining → Risk inference: provides model
- User → Web interface: uses
- Web interface → Analysis panel: displays
- Web interface → Alert banner: displays
- Web interface → Chat panel: displays
- Analysis panel → Analysis API: requests analysis
- Analysis API → Analysis queries: uses queries
- Analysis API → KPI calculations: calculates KPIs
- Chat panel → Chatbot API: sends questions
- Chatbot API → Chatbot chain: dispatches
- Chatbot chain → Data to text: formats data
- Chatbot chain → Vector store setup: uses retrieval