How hnawaz007/pythondataanalysis works
This repository is a collection of independent Python examples for data engineering, analytics, machine learning, and web applications, rather than one integrated product. A prominent workflow extracts data from SQL or object storage, transforms or selects it, and loads it into a database or lake; separate applications query stored data and present dashboards or natural-language answers. The tree also documents Airflow, Dagster, dbt, and cloud examples. Sampled source supports the detailed ETL and application links shown; other listed examples are represented as standalone branches where their wiring was not sampled.
Main entry points: NLP & AI database integration/app.py Database chat app · build_etl_pipeline_python.py SQL ETL example · main.py AWS lake loader · VoiceAssistant/app.py Voice assistant
Components 23
Data pipelines
- SQL ETL example
ETL Pipeline/build_etl_pipeline_python.py - AWS lake loader
AWS Data Lake/main.py - SQL dlt pipelines
ETL Pipeline/dltproject/sql_database_pipeline.py - Chess dlt source
ETL Pipeline/dltproject/chess_pipeline.py - SQL Server stream
ETL Pipeline/dltproject/data_pipeline.py - S3 sales loader
dbt-datapond/etl_pipeline.py
Orchestration
- Airflow DAG examples
AirflowSession2/dags/airbyte_dag.py - SQL DAG examples
AirflowSession2/dags/run_sql_dag.py - Dagster ETL examples
Dagster/etl/etl/jobs/run_etl.py - Dagster data assets
dagster-project/etl/etl/assets_dataanalysis.py
Analytics apps
- Sales dashboard
Clickhouse/app.py
AI assistants
- Database chat app
NLP & AI database integration/app.py - Voice assistant
VoiceAssistant/app.py
Other components
- Data engineer
- SQL Server
- S3 / MinIO
- PostgreSQL
- ClickHouse
- App user
- SQL-generating model
- Answer-generating model
- Microphone input
- Speech and web services
Connections 23
- Data engineer → SQL ETL example: runs extraction
- SQL ETL example → SQL Server: reads tables
- SQL ETL example → PostgreSQL: writes tables
- Data engineer → AWS lake loader: runs extraction
- AWS lake loader → SQL Server: reads tables
- AWS lake loader → S3 / MinIO: uploads CSV
- Data engineer → SQL dlt pipelines: runs pipeline
- SQL dlt pipelines → PostgreSQL: loads tables
- Data engineer → Chess dlt source: runs pipeline
- Chess dlt source → PostgreSQL: loads resources
- SQL Server stream → SQL Server: streams rows
- SQL Server stream → PostgreSQL: loads rows
- S3 sales loader → S3 / MinIO: reads parquet
- S3 sales loader → PostgreSQL: loads records
- App user → Database chat app: asks question
- Database chat app → SQL-generating model: generates SQL
- Database chat app → PostgreSQL: runs query
- Database chat app → Answer-generating model: formats answer
- App user → Sales dashboard: views dashboard
- Sales dashboard → ClickHouse: queries sales
- Microphone input → Voice assistant: provides audio
- Voice assistant → Speech and web services: recognizes and speaks
- Voice assistant → App user: opens web result