How zain-ul-abdin45/analytics-engineer-project works
Project Type and Purpose
• It is not a traditional web app but an end-to-end data pipeline with four layers: Ingestion, Processing, Storage, and Visualization.
File Structure Analysis
• Top level:
• data/ ─ raw data design docs and (presumably) raw data files (Parquet, JSON).
• docs/ ─ design docs and PDF of final dashboards.
• src/ ─ Python scripts and Jupyter notebook for ETL and EDA.
• README.md, requirements.txt.
• ETL scripts in src:
• ETL_client_part_collection_data_processing_scripts.py
• ETL_data_processing_sonar_results.py
• EDA notebook in src:
• EDA_Markt_Pilot.ipynb
Architecture Overview
The pipeline comprises four main components:
A. Raw Data Sources
B. ETL & EDA Processing
C. Relational Storage
D. Dashboarding & Visualization
Components and Interactions
a. Raw Data Sources (Layer 1)
• Files in data/: Parquet files with nested JSON fields (e.g., sonar_results).
• Technology: DuckDB for on-the-fly querying, minimal memory.
b. ETL & EDA Processing (Layer 2)
• Two Python ETL scripts that:
• Use pandas + duckdb to extract and normalize nested JSON.
• Clean, standardize, and transform data.
• Load into PostgreSQL via SQLAlchemy/psycopg2.
• One Jupyter notebook (EDA_Markt_Pilot.ipynb) that:
• Reads cleaned data (either via DuckDB or PostgreSQL).
• Performs exploratory analyses with pandas, seaborn, matplotlib.
• Execution: Manual or scheduled via cron/CI (not explicitly in repo).
c. Relational Storage (Layer 3)
• PostgreSQL instance running locally.
• Schema mirrors cleaned data: tables for parts, shops, prices, delivery_times, etc.
• Indexed for query performance.
d. Dashboarding & Visualization (Layer 4)
• Apache Superset connected to PostgreSQL.
• Dashboards: country distributions, price trends, delivery-time histograms, completeness heatmaps.
• Final dashboards exported as PDF (in docs/).
Architectural Patterns & Principles
• Layered architecture (Ingestion → Processing → Storage → Presentation).
• Separation of concerns:
• ETL scripts are distinct from EDA notebooks.
• Data storage is decoupled from visualization.
• Use of lightweight in-process querying (DuckDB) for speed, then durable storage in PostgreSQL.
• Modular: each script/notebook handles a single responsibility.