# honey-singh716/dataengineeringproject architecture

> Architecture diagram of the GitHub repository honey-singh716/dataengineeringproject, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/honey-singh716/dataengineeringproject
- Repository: https://github.com/honey-singh716/dataengineeringproject

## Overview

RoadInsight India turns accident datasets into a PostgreSQL analytics store and Streamlit dashboards. The documented data workflow extracts and cleans CSVs, loads normalized tables, and supports national, geographic, temporal, accident, vehicle, and severity analysis. A separate AI Analyst workflow accepts a question in the dashboard, generates SQL through Groq, validates it, runs it using a read-only database connection, and presents results. ETL internals and most dashboard sections were not sampled, so those documented capabilities are represented without asserting unverified internal calls.

Main entry points: [`ai_analyst/services/text_to_sql_service.py`](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/services/text_to_sql_service.py) (Text-to-SQL Service), [`ai_analyst/database/query_executor.py`](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/database/query_executor.py) (Query Executor), [`dashboard/app.py`](https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py) (Overview Trends), [`ai_analyst/llm/sql_generator.py`](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/llm/sql_generator.py) (SQL Generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data Pipeline"]
  node_raw["Raw CSVs"]
  node_clean["Cleaned Datasets"]
  node_etl["ETL Loader<br/>[06_etl_loader.py]"]
  node_schema["Schema Setup"]
end

subgraph group_analytics["Analytics Store"]
  node_db[("PostgreSQL")]
end

subgraph group_dashboard["Dashboard"]
  node_overview["Overview Trends<br/>[app.py]"]
  node_geo["Geographic Analysis<br/>[app.py]"]
  node_time["Temporal Analysis<br/>[app.py]"]
  node_accident["Accident Analysis<br/>[app.py]"]
  node_vehicle["Vehicle Analysis<br/>[app.py]"]
  node_severity["Severity Analysis<br/>[app.py]"]
  node_ai_tab["AI Analyst UI<br/>[ai_analyst_tab.py]"]
end

subgraph group_ai["AI Analyst"]
  node_service["Text-to-SQL Service"]
  node_generator["SQL Generator<br/>[sql_generator.py]"]
  node_prompts["Prompt Templates"]
  node_groq["Groq API Client<br/>[groq_client.py]"]
end

subgraph group_safety["Query Safety"]
  node_schema_context["Schema Context<br/>[schema_context.py]"]
  node_validator["SQL Validator<br/>[sql_validator.py]"]
  node_allowlist["Table Allowlist<br/>[table_allowlist.py]"]
  node_executor["Query Executor<br/>[query_executor.py]"]
  node_connection["Read-only Connection<br/>[connection.py]"]
  node_config["Runtime Settings<br/>[config.py]"]
end

node_operator(("Dashboard User"))
node_groq_api{{"Groq Service"}}

node_raw -->|"extracts"| node_etl
node_clean -->|"loads from"| node_etl
node_etl -->|"loads transformed data"| node_db
node_schema -->|"creates schema"| node_db
node_operator -.->|"views"| node_overview
node_overview -->|"queries"| node_db
node_geo -.->|"queries"| node_db
node_time -.->|"queries"| node_db
node_accident -.->|"queries"| node_db
node_vehicle -.->|"queries"| node_db
node_severity -.->|"queries"| node_db
node_overview -->|"renders"| node_ai_tab
node_operator -.->|"asks"| node_ai_tab
node_ai_tab -->|"submits question"| node_service
node_service -->|"generates SQL"| node_generator
node_service -->|"validates SQL"| node_validator
node_service -->|"executes validated SQL"| node_executor
node_generator -->|"builds prompts"| node_prompts
node_generator -->|"requests completion"| node_groq
node_groq -->|"calls"| node_groq_api
node_prompts -->|"uses schema"| node_schema_context
node_validator -->|"checks tables"| node_allowlist
node_executor -->|"gets engine"| node_connection
node_executor -->|"runs query"| node_db
node_connection -->|"connects read-only"| node_db
node_connection -->|"reads settings"| node_config
node_executor -->|"uses limits"| node_config

click node_raw "https://github.com/honey-singh716/dataengineeringproject/tree/main/Data"
click node_clean "https://github.com/honey-singh716/dataengineeringproject/tree/main/DataSets"
click node_etl "https://github.com/honey-singh716/dataengineeringproject/blob/main/db_design/06_etl_loader.py"
click node_schema "https://github.com/honey-singh716/dataengineeringproject/tree/main/db_design/sql"
click node_overview "https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py"
click node_geo "https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py"
click node_time "https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py"
click node_accident "https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py"
click node_vehicle "https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py"
click node_severity "https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py"
click node_ai_tab "https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/ai_analyst_tab.py"
click node_service "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/services/text_to_sql_service.py"
click node_generator "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/llm/sql_generator.py"
click node_prompts "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/llm/prompt_templates.py"
click node_groq "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/llm/groq_client.py"
click node_schema_context "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/database/schema_context.py"
click node_validator "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/security/sql_validator.py"
click node_allowlist "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/security/table_allowlist.py"
click node_executor "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/database/query_executor.py"
click node_connection "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/database/connection.py"
click node_config "https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw,node_clean,node_etl,node_schema,node_operator toneBlue
class node_db toneAmber
class node_overview,node_geo,node_time,node_accident,node_vehicle,node_severity,node_ai_tab toneMint
class node_service,node_generator,node_prompts,node_groq toneRose
class node_schema_context,node_validator,node_allowlist,node_executor,node_connection,node_config,node_groq_api toneIndigo
```

## Components

### Data Pipeline

- [Raw CSVs](https://github.com/honey-singh716/dataengineeringproject/tree/main/Data): `Data`
- [Cleaned Datasets](https://github.com/honey-singh716/dataengineeringproject/tree/main/DataSets): `DataSets`
- [ETL Loader](https://github.com/honey-singh716/dataengineeringproject/blob/main/db_design/06_etl_loader.py): `db_design/06_etl_loader.py`
- [Schema Setup](https://github.com/honey-singh716/dataengineeringproject/tree/main/db_design/sql): `db_design/sql`

### Analytics Store

- **PostgreSQL**

### Dashboard

- [Overview Trends](https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py): `dashboard/app.py`
- [Geographic Analysis](https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py): `dashboard/app.py`
- [Temporal Analysis](https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py): `dashboard/app.py`
- [Accident Analysis](https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py): `dashboard/app.py`
- [Vehicle Analysis](https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py): `dashboard/app.py`
- [Severity Analysis](https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/app.py): `dashboard/app.py`
- [AI Analyst UI](https://github.com/honey-singh716/dataengineeringproject/blob/main/dashboard/ai_analyst_tab.py): `dashboard/ai_analyst_tab.py`

### AI Analyst

- [Text-to-SQL Service](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/services/text_to_sql_service.py): `ai_analyst/services/text_to_sql_service.py`
- [SQL Generator](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/llm/sql_generator.py): `ai_analyst/llm/sql_generator.py`
- [Prompt Templates](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/llm/prompt_templates.py): `ai_analyst/llm/prompt_templates.py`
- [Groq API Client](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/llm/groq_client.py): `ai_analyst/llm/groq_client.py`

### Query Safety

- [Schema Context](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/database/schema_context.py): `ai_analyst/database/schema_context.py`
- [SQL Validator](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/security/sql_validator.py): `ai_analyst/security/sql_validator.py`
- [Table Allowlist](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/security/table_allowlist.py): `ai_analyst/security/table_allowlist.py`
- [Query Executor](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/database/query_executor.py): `ai_analyst/database/query_executor.py`
- [Read-only Connection](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/database/connection.py): `ai_analyst/database/connection.py`
- [Runtime Settings](https://github.com/honey-singh716/dataengineeringproject/blob/main/ai_analyst/config.py): `ai_analyst/config.py`

### Other components

- **Dashboard User**
- **Groq Service**

## Connections

- Raw CSVs → ETL Loader: extracts (evidence: `README.md`)
- Cleaned Datasets → ETL Loader: loads from (evidence: `README.md`)
- ETL Loader → PostgreSQL: loads transformed data (evidence: `README.md`)
- Schema Setup → PostgreSQL: creates schema (evidence: `README.md`)
- Dashboard User → Overview Trends: views
- Overview Trends → PostgreSQL: queries (evidence: `dashboard/app.py`)
- Geographic Analysis → PostgreSQL: queries
- Temporal Analysis → PostgreSQL: queries
- Accident Analysis → PostgreSQL: queries
- Vehicle Analysis → PostgreSQL: queries
- Severity Analysis → PostgreSQL: queries
- Overview Trends → AI Analyst UI: renders (evidence: `dashboard/app.py`)
- Dashboard User → AI Analyst UI: asks
- AI Analyst UI → Text-to-SQL Service: submits question (evidence: `dashboard/ai_analyst_tab.py`)
- Text-to-SQL Service → SQL Generator: generates SQL (evidence: `ai_analyst/services/text_to_sql_service.py`)
- Text-to-SQL Service → SQL Validator: validates SQL (evidence: `ai_analyst/services/text_to_sql_service.py`)
- Text-to-SQL Service → Query Executor: executes validated SQL (evidence: `ai_analyst/services/text_to_sql_service.py`)
- SQL Generator → Prompt Templates: builds prompts (evidence: `ai_analyst/llm/sql_generator.py`)
- SQL Generator → Groq API Client: requests completion (evidence: `ai_analyst/llm/sql_generator.py`)
- Groq API Client → Groq Service: calls (evidence: `ai_analyst/llm/groq_client.py`)
- Prompt Templates → Schema Context: uses schema (evidence: `ai_analyst/llm/prompt_templates.py`)
- SQL Validator → Table Allowlist: checks tables (evidence: `ai_analyst/security/sql_validator.py`)
- Query Executor → Read-only Connection: gets engine (evidence: `ai_analyst/database/query_executor.py`)
- Query Executor → PostgreSQL: runs query (evidence: `ai_analyst/database/query_executor.py`)
- Read-only Connection → PostgreSQL: connects read-only (evidence: `ai_analyst/database/connection.py`)
- Read-only Connection → Runtime Settings: reads settings (evidence: `ai_analyst/database/connection.py`)
- Query Executor → Runtime Settings: uses limits (evidence: `ai_analyst/database/query_executor.py`)