# fhrizwi/celebal_technologies_internship architecture

> Architecture diagram of the GitHub repository fhrizwi/celebal_technologies_internship, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fhrizwi/celebal_technologies_internship
- Repository: https://github.com/fhrizwi/celebal_technologies_internship

## Overview

1. **Overall shape:** This repository is not a single application; it is a collection of **self-contained weekly data engineering assignments**. Each week has its own inputs, code/notebook, outputs, and documentation. There is **no shared runtime or shared pipeline** across weeks.

2. **Primary boundary:** The repo splits into two kinds of assets:
   - **Executable artifacts**: Python scripts, Jupyter notebooks, SQL files.
   - **Documentation/evidence**: READMEs, PDFs, screenshots, and answer text files.
   The documentation is important for assessment, but it is not part of the data-processing runtime.

3. **Week 1 — Pandas cleaning pipeline:** A local notebook-based workflow ingests `Combined_dataset.csv`, explores and cleans the data, handles missing values/duplicates, and writes out `cleaned_dataset.csv`. This is a classic **single-machine ETL step** using Python + Pandas.

4. **Week 2 — MySQL e-commerce analysis:** This module is a **SQL-only analytics layer** centered on `all_queries.sql`. The workload appears to run against a MySQL database and produces business insights plus screenshots. The logic is analytical rather than data-moving: filter, aggregate, join, and transaction-based queries over e-commerce data.

5. **Week 3 — Advanced SQL on Superstore data:** This is another database analytics module, but organized as a staged SQL workflow:
   - table creation
   - data insertion
   - subqueries
   - CTEs
   - window functions
   - final analysis
   The data source is the `Sample - Superstore.csv` file, and the architecture is a **SQL transformation/analysis pipeline** inside the database.

6. **Week 4 — Azure Data Factory pipeline:** This is the only clearly cloud-orchestrated pipeline. The flow is:
   **Source Blob → Get Metadata → Copy Data → Destination Blob**
   It uses **Azure Blob Storage** as source and sink, with **Azure Data Factory** as the orchestration layer. IAM/linked services/datasets are the main infrastructure boundaries here.

7. **Week 5 — PySpark batch processing:** A local Spark job (`spark_assignment.py`) reads `sales.csv`, performs cleaning/transformation/aggregation, and runs against a Python + PySpark environment defined by `requirements.txt`. This is a **distributed-data processing job**, but executed as a local batch workflow rather than a cluster-managed service in the repo.

8. **Week 6 — Spark read/write pipeline with multiple outputs:** `main.py` processes `source.csv` and writes results to both **CSV** and **Parquet** under `output/`. This week adds an explicit **format conversion boundary**: the same transformed dataset is persisted in two storage formats, showing Spark as an extraction/transformation/load layer.

9. **Week 7 — Notebook-based Delta Lake assignment:** This week is organized around a Jupyter notebook and a source CSV (`sample-store.csv`), producing a cleaned output file. Architecturally, it follows the same pattern as the other data-prep weeks: **file input → notebook transformations → cleaned file output**, with notebook execution as the main control plane.

10. **Cross-cutting data flow pattern:** Across the repo, the dominant pattern is **CSV-in / transformed output-out**, with occasional database and cloud storage destinations. The main storage/media types are:
   - CSV inputs and outputs
   - SQL scripts and database tables
   - Parquet output for Spark
   - Azure Blob Storage for cloud ETL
   - Notebook/PDF/screenshot artifacts for reporting

11. **Tooling and runtime boundaries:** The main execution environments are:
   - **Jupyter notebooks** for exploratory cleaning and notebook assignments
   - **Python scripts** for repeatable local batch processing
   - **MySQL Workbench / SQL engine** for database analytics
   - **Azure Data Factory + Blob Storage** for managed cloud orchestration
   - **PySpark** for scalable batch transformations

12. **How to diagram it:** The cleanest architecture diagram is a **portfolio of independent pipelines**, not a monolith. Show each week as a separate lane with its own source data, processing engine, and output artifacts, and highlight the one cloud-native lane in Week 4 plus the Spark batch lanes in Weeks 5–7.

Main entry points: [`WEEK_6/Spark_Assignment/main.py`](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_6/Spark_Assignment/main.py) (Spark flow), [`WEEK_1/notebook/data_cleaning.ipynb`](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_1/notebook/data_cleaning.ipynb) (Cleaning notebook), [`WEEK_3/Superstore-SQL-Analysis/sql/01_create_tables.sql`](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_3/Superstore-SQL-Analysis/sql/01_create_tables.sql) (SQL stages), [`WEEK_4/Azure-ADF-Data-Pipeline/README.md`](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_4/Azure-ADF-Data-Pipeline/README.md) (ADF pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_week1["Week 1"]
  node_w1_input["Raw CSV<br/>source data"]
  node_w1_nb["Cleaning notebook<br/>pandas notebook"]
  node_w1_output["Cleaned CSV<br/>cleaned output"]
end

subgraph group_week2["Week 2"]
  node_w2_sql["Query script<br/>sql analytics<br/>[all_queries.sql]"]
  node_w2_answers["Insights<br/>evidence<br/>[insights.txt]"]
end

subgraph group_week3["Week 3"]
  node_w3_input["Superstore CSV<br/>source data"]
  node_w3_sql["SQL stages<br/>db workflow"]
  node_w3_report["Results<br/>analysis evidence"]
end

subgraph group_week4["Week 4"]
  node_w4_source["Blob source<br/>cloud storage"]
  node_w4_adf["ADF pipeline<br/>cloud orchestrator<br/>[README.md]"]
  node_w4_dest["Blob sink<br/>cloud storage"]
end

subgraph group_week5["Week 5"]
  node_w5_input["Sales CSV<br/>source data<br/>[sales.csv]"]
  node_w5_spark["Spark job<br/>pyspark batch"]
  node_w5_docs["QA docs<br/>evidence"]
end

subgraph group_week6["Week 6"]
  node_w6_input["Source CSV<br/>source data<br/>[source.csv]"]
  node_w6_main["Spark flow<br/>pyspark batch<br/>[main.py]"]
  node_w6_csv["CSV output<br/>persisted output"]
  node_w6_parquet[("Parquet output<br/>persisted output")]
end

subgraph group_week7["Week 7"]
  node_w7_input["Store CSV<br/>source data<br/>[sample-store.csv]"]
  node_w7_nb["Delta notebook<br/>notebook ETL"]
  node_w7_output["Cleaned CSV<br/>cleaned output<br/>[cleaned_store.csv]"]
end

node_w1_input -->|"read"| node_w1_nb
node_w1_nb -->|"write"| node_w1_output
node_w2_sql -->|"derive"| node_w2_answers
node_w3_input -->|"load"| node_w3_sql
node_w3_sql -->|"document"| node_w3_report
node_w4_source -->|"ingest"| node_w4_adf
node_w4_adf -->|"copy"| node_w4_dest
node_w5_input -->|"process"| node_w5_spark
node_w5_spark -->|"evidence"| node_w5_docs
node_w6_input -->|"process"| node_w6_main
node_w6_main -->|"write"| node_w6_csv
node_w6_main -->|"write"| node_w6_parquet
node_w7_input -->|"read"| node_w7_nb
node_w7_nb -->|"write"| node_w7_output

click node_w1_input "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_1/data/Combined_dataset.csv"
click node_w1_nb "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_1/notebook/data_cleaning.ipynb"
click node_w1_output "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_1/data/cleaned_dataset.csv"
click node_w2_sql "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_2/SQL_Ecommerce_Analysis/queries/all_queries.sql"
click node_w2_answers "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_2/insights.txt"
click node_w3_input "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_3/Superstore-SQL-Analysis/data/Sample - Superstore.csv"
click node_w3_sql "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_3/Superstore-SQL-Analysis/sql/01_create_tables.sql"
click node_w3_report "https://github.com/fhrizwi/celebal_technologies_internship/tree/main/WEEK_3/Superstore-SQL-Analysis/screenshots"
click node_w4_source "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_4/Azure-ADF-Data-Pipeline/Dataset/Sample-Superstore.csv"
click node_w4_adf "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_4/Azure-ADF-Data-Pipeline/README.md"
click node_w4_dest "https://github.com/fhrizwi/celebal_technologies_internship/tree/main/WEEK_4/Azure-ADF-Data-Pipeline/Screenshots"
click node_w5_input "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_5/Spark-Assignment/data/sales.csv"
click node_w5_spark "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_5/Spark-Assignment/spark_assignment.py"
click node_w5_docs "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_5/Spark-Assignment/Apache_Spark_Week-5_QA.pdf"
click node_w6_input "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_6/Spark_Assignment/data/source.csv"
click node_w6_main "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_6/Spark_Assignment/main.py"
click node_w6_csv "https://github.com/fhrizwi/celebal_technologies_internship/tree/main/WEEK_6/Spark_Assignment/output/csv_output"
click node_w6_parquet "https://github.com/fhrizwi/celebal_technologies_internship/tree/main/WEEK_6/Spark_Assignment/output/parquet_data"
click node_w7_input "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_7/DeltaLakeAssignment/data/sample-store.csv"
click node_w7_nb "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_7/DeltaLakeAssignment/notebook/DeltaLakeAssignment.ipynb"
click node_w7_output "https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_7/DeltaLakeAssignment/output/cleaned_store.csv"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_w1_input,node_w1_nb,node_w1_output,node_w7_input,node_w7_nb,node_w7_output toneBlue
class node_w2_sql,node_w2_answers toneAmber
class node_w3_input,node_w3_sql,node_w3_report toneMint
class node_w4_source,node_w4_adf,node_w4_dest toneRose
class node_w5_input,node_w5_spark,node_w5_docs toneIndigo
class node_w6_input,node_w6_main,node_w6_csv,node_w6_parquet toneTeal
```

## Components

### Week 1

Pandas notebook ETL

- [Raw CSV](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_1/data/Combined_dataset.csv): source data · Combined dataset input for cleaning · `WEEK_1/data/Combined_dataset.csv`
- [Cleaning notebook](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_1/notebook/data_cleaning.ipynb): pandas notebook · Explores and cleans the dataset · `WEEK_1/notebook/data_cleaning.ipynb`
- [Cleaned CSV](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_1/data/cleaned_dataset.csv): cleaned output · Normalized dataset written by the notebook · `WEEK_1/data/cleaned_dataset.csv`

### Week 2

MySQL e-commerce analysis

- [Query script](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_2/SQL_Ecommerce_Analysis/queries/all_queries.sql): sql analytics · E-commerce analysis queries and transactional SQL · `WEEK_2/SQL_Ecommerce_Analysis/queries/all_queries.sql`
- [Insights](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_2/insights.txt): evidence · Written answers and screenshots supporting analysis · `WEEK_2/insights.txt`

### Week 3

Superstore SQL workflow

- [Superstore CSV](https://github.com/fhrizwi/celebal_technologies_internship/blob/HEAD/WEEK_3/Superstore-SQL-Analysis/data/Sample%20-%20Superstore.csv): source data · Raw Superstore dataset · `WEEK_3/Superstore-SQL-Analysis/data/Sample - Superstore.csv`
- [SQL stages](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_3/Superstore-SQL-Analysis/sql/01_create_tables.sql): db workflow · Table creation, load, subqueries, CTEs, window functions, final analysis · `WEEK_3/Superstore-SQL-Analysis/sql/01_create_tables.sql`
- [Results](https://github.com/fhrizwi/celebal_technologies_internship/tree/main/WEEK_3/Superstore-SQL-Analysis/screenshots): analysis evidence · Screenshots and narrative outputs for the SQL analysis · `WEEK_3/Superstore-SQL-Analysis/screenshots`

### Week 4

Azure Data Factory pipeline

- [Blob source](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_4/Azure-ADF-Data-Pipeline/Dataset/Sample-Superstore.csv): cloud storage · Azure Blob input dataset · `WEEK_4/Azure-ADF-Data-Pipeline/Dataset/Sample-Superstore.csv`
- [ADF pipeline](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_4/Azure-ADF-Data-Pipeline/README.md): cloud orchestrator · Get metadata and copy orchestration in Azure Data Factory · `WEEK_4/Azure-ADF-Data-Pipeline/README.md`
- [Blob sink](https://github.com/fhrizwi/celebal_technologies_internship/tree/main/WEEK_4/Azure-ADF-Data-Pipeline/Screenshots): cloud storage · Destination blob storage target · `WEEK_4/Azure-ADF-Data-Pipeline/Screenshots`

### Week 5

PySpark batch job

- [Sales CSV](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_5/Spark-Assignment/data/sales.csv): source data · Input sales dataset · `WEEK_5/Spark-Assignment/data/sales.csv`
- [Spark job](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_5/Spark-Assignment/spark_assignment.py): pyspark batch · Python Spark transformation and aggregation script · `WEEK_5/Spark-Assignment/spark_assignment.py`
- [QA docs](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_5/Spark-Assignment/Apache_Spark_Week-5_QA.pdf): evidence · Assignment PDFs and supporting material · `WEEK_5/Spark-Assignment/Apache_Spark_Week-5_QA.pdf`

### Week 6

Spark dual-format output

- [Source CSV](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_6/Spark_Assignment/data/source.csv): source data · Input file for Spark processing · `WEEK_6/Spark_Assignment/data/source.csv`
- [Spark flow](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_6/Spark_Assignment/main.py): pyspark batch · Transforms data and writes multiple output formats · `WEEK_6/Spark_Assignment/main.py`
- [CSV output](https://github.com/fhrizwi/celebal_technologies_internship/tree/main/WEEK_6/Spark_Assignment/output/csv_output): persisted output · Spark output written as CSV · `WEEK_6/Spark_Assignment/output/csv_output`
- [Parquet output](https://github.com/fhrizwi/celebal_technologies_internship/tree/main/WEEK_6/Spark_Assignment/output/parquet_data): persisted output · Spark output written as Parquet · `WEEK_6/Spark_Assignment/output/parquet_data`

### Week 7

Delta Lake notebook assignment

- [Store CSV](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_7/DeltaLakeAssignment/data/sample-store.csv): source data · Notebook input dataset · `WEEK_7/DeltaLakeAssignment/data/sample-store.csv`
- [Delta notebook](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_7/DeltaLakeAssignment/notebook/DeltaLakeAssignment.ipynb): notebook ETL · Notebook-based cleaning and transformation · `WEEK_7/DeltaLakeAssignment/notebook/DeltaLakeAssignment.ipynb`
- [Cleaned CSV](https://github.com/fhrizwi/celebal_technologies_internship/blob/main/WEEK_7/DeltaLakeAssignment/output/cleaned_store.csv): cleaned output · Processed store dataset written out · `WEEK_7/DeltaLakeAssignment/output/cleaned_store.csv`

## Connections

- Raw CSV → Cleaning notebook: read · Notebook loads raw CSV for inspection and cleaning
- Cleaning notebook → Cleaned CSV: write · Cleansed dataset persisted to CSV
- Query script → Insights: derive · Query results summarized as insights and evidence
- Superstore CSV → SQL stages: load · Superstore data is staged into SQL analysis steps
- SQL stages → Results: document · Final analysis is captured in screenshots and outputs
- Blob source → ADF pipeline: ingest · ADF reads from the source blob
- ADF pipeline → Blob sink: copy · Pipeline moves data to destination blob
- Sales CSV → Spark job: process · PySpark job reads and transforms sales data
- Spark job → QA docs: evidence · Supporting assignment documentation accompanies the job
- Source CSV → Spark flow: process · Spark job consumes the source CSV
- Spark flow → CSV output: write · Transformed data persisted as CSV
- Spark flow → Parquet output: write · Transformed data persisted as Parquet
- Store CSV → Delta notebook: read · Notebook ingests the store CSV
- Delta notebook → Cleaned CSV: write · Cleaned output written back to disk
