honey-singh716/dataengineeringproject

How honey-singh716/dataengineeringproject works

RoadInsight India turns accident datasets into a PostgreSQL analytics store and Streamlit dashboards. The documented data workflow extracts and cleans CSVs, loads normalized tables, and supports national, geographic, temporal, accident, vehicle, and severity analysis. A separate AI Analyst workflow accepts a question in the dashboard, generates SQL through Groq, validates it, runs it using a read-only database connection, and presents results. ETL internals and most dashboard sections were not sampled, so those documented capabilities are represented without asserting unverified internal calls.

Main entry points: text_to_sql_service.py Text-to-SQL Service · query_executor.py Query Executor · app.py Overview Trends · sql_generator.py SQL Generator

Components 24

Data Pipeline

Raw CSVs Data
Cleaned Datasets DataSets
ETL Loader db_design/06_etl_loader.py
Schema Setup db_design/sql

Analytics Store

PostgreSQL

Dashboard

Overview Trends dashboard/app.py
Geographic Analysis dashboard/app.py
Temporal Analysis dashboard/app.py
Accident Analysis dashboard/app.py
Vehicle Analysis dashboard/app.py
Severity Analysis dashboard/app.py
AI Analyst UI dashboard/ai_analyst_tab.py

AI Analyst

Text-to-SQL Service ai_analyst/services/text_to_sql_service.py
SQL Generator ai_analyst/llm/sql_generator.py
Prompt Templates ai_analyst/llm/prompt_templates.py
Groq API Client ai_analyst/llm/groq_client.py

Query Safety

Schema Context ai_analyst/database/schema_context.py
SQL Validator ai_analyst/security/sql_validator.py
Table Allowlist ai_analyst/security/table_allowlist.py
Query Executor ai_analyst/database/query_executor.py
Read-only Connection ai_analyst/database/connection.py
Runtime Settings ai_analyst/config.py

Other components

Dashboard User
Groq Service

Connections 27

Raw CSVs → ETL Loader: extracts README.md
Cleaned Datasets → ETL Loader: loads from README.md
ETL Loader → PostgreSQL: loads transformed data README.md
Schema Setup → PostgreSQL: creates schema README.md
Dashboard User → Overview Trends: views
Overview Trends → PostgreSQL: queries dashboard/app.py
Geographic Analysis → PostgreSQL: queries
Temporal Analysis → PostgreSQL: queries
Accident Analysis → PostgreSQL: queries
Vehicle Analysis → PostgreSQL: queries
Severity Analysis → PostgreSQL: queries
Overview Trends → AI Analyst UI: renders dashboard/app.py
Dashboard User → AI Analyst UI: asks
AI Analyst UI → Text-to-SQL Service: submits question dashboard/ai_analyst_tab.py
Text-to-SQL Service → SQL Generator: generates SQL ai_analyst/services/text_to_sql_service.py
Text-to-SQL Service → SQL Validator: validates SQL ai_analyst/services/text_to_sql_service.py
Text-to-SQL Service → Query Executor: executes validated SQL ai_analyst/services/text_to_sql_service.py
SQL Generator → Prompt Templates: builds prompts ai_analyst/llm/sql_generator.py
SQL Generator → Groq API Client: requests completion ai_analyst/llm/sql_generator.py
Groq API Client → Groq Service: calls ai_analyst/llm/groq_client.py
Prompt Templates → Schema Context: uses schema ai_analyst/llm/prompt_templates.py
SQL Validator → Table Allowlist: checks tables ai_analyst/security/sql_validator.py
Query Executor → Read-only Connection: gets engine ai_analyst/database/query_executor.py
Query Executor → PostgreSQL: runs query ai_analyst/database/query_executor.py
Read-only Connection → PostgreSQL: connects read-only ai_analyst/database/connection.py
Read-only Connection → Runtime Settings: reads settings ai_analyst/database/connection.py
Query Executor → Runtime Settings: uses limits ai_analyst/database/query_executor.py

Drawn by GitDiagram on November 15, 2026 from the repository's file tree, README and source files. Read as Markdown ·