How fhrizwi/celebal_technologies_internship works
1. Overall shape: This repository is not a single application; it is a collection of self-contained weekly data engineering assignments. Each week has its own inputs, code/notebook, outputs, and documentation. There is no shared runtime or shared pipeline across weeks.
2. Primary boundary: The repo splits into two kinds of assets:
- Executable artifacts: Python scripts, Jupyter notebooks, SQL files.
- Documentation/evidence: READMEs, PDFs, screenshots, and answer text files.
The documentation is important for assessment, but it is not part of the data-processing runtime.
3. Week 1 — Pandas cleaning pipeline: A local notebook-based workflow ingests Combined_dataset.csv, explores and cleans the data, handles missing values/duplicates, and writes out cleaned_dataset.csv. This is a classic single-machine ETL step using Python + Pandas.
4. Week 2 — MySQL e-commerce analysis: This module is a SQL-only analytics layer centered on all_queries.sql. The workload appears to run against a MySQL database and produces business insights plus screenshots. The logic is analytical rather than data-moving: filter, aggregate, join, and transaction-based queries over e-commerce data.
5. Week 3 — Advanced SQL on Superstore data: This is another database analytics module, but organized as a staged SQL workflow:
- table creation
- data insertion
- subqueries
- CTEs
- window functions
- final analysis
The data source is the Sample - Superstore.csv file, and the architecture is a SQL transformation/analysis pipeline inside the database.
6. Week 4 — Azure Data Factory pipeline: This is the only clearly cloud-orchestrated pipeline. The flow is:
Source Blob → Get Metadata → Copy Data → Destination Blob
It uses Azure Blob Storage as source and sink, with Azure Data Factory as the orchestration layer. IAM/linked services/datasets are the main infrastructure boundaries here.
7. Week 5 — PySpark batch processing: A local Spark job (spark_assignment.py) reads sales.csv, performs cleaning/transformation/aggregation, and runs against a Python + PySpark environment defined by requirements.txt. This is a distributed-data processing job, but executed as a local batch workflow rather than a cluster-managed service in the repo.
8. Week 6 — Spark read/write pipeline with multiple outputs: main.py processes source.csv and writes results to both CSV and Parquet under output/. This week adds an explicit format conversion boundary: the same transformed dataset is persisted in two storage formats, showing Spark as an extraction/transformation/load layer.
9. Week 7 — Notebook-based Delta Lake assignment: This week is organized around a Jupyter notebook and a source CSV (sample-store.csv), producing a cleaned output file. Architecturally, it follows the same pattern as the other data-prep weeks: file input → notebook transformations → cleaned file output, with notebook execution as the main control plane.
10. Cross-cutting data flow pattern: Across the repo, the dominant pattern is CSV-in / transformed output-out, with occasional database and cloud storage destinations. The main storage/media types are:
- CSV inputs and outputs
- SQL scripts and database tables
- Parquet output for Spark
- Azure Blob Storage for cloud ETL
- Notebook/PDF/screenshot artifacts for reporting
11. Tooling and runtime boundaries: The main execution environments are:
- Jupyter notebooks for exploratory cleaning and notebook assignments
- Python scripts for repeatable local batch processing
- MySQL Workbench / SQL engine for database analytics
- Azure Data Factory + Blob Storage for managed cloud orchestration
- PySpark for scalable batch transformations
12. How to diagram it: The cleanest architecture diagram is a portfolio of independent pipelines, not a monolith. Show each week as a separate lane with its own source data, processing engine, and output artifacts, and highlight the one cloud-native lane in Week 4 plus the Spark batch lanes in Weeks 5–7.
Main entry points: main.py Spark flow · data_cleaning.ipynb Cleaning notebook · 01_create_tables.sql SQL stages · README.md ADF pipeline
Components 21
Week 1
Pandas notebook ETL
- Raw CSV · Combined dataset input for cleaning
WEEK_1/data/Combined_dataset.csv - Cleaning notebook · Explores and cleans the dataset
WEEK_1/notebook/data_cleaning.ipynb - Cleaned CSV · Normalized dataset written by the notebook
WEEK_1/data/cleaned_dataset.csv
Week 2
MySQL e-commerce analysis
- Query script · E-commerce analysis queries and transactional SQL
WEEK_2/SQL_Ecommerce_Analysis/queries/all_queries.sql - Insights · Written answers and screenshots supporting analysis
WEEK_2/insights.txt
Week 3
Superstore SQL workflow
- Superstore CSV · Raw Superstore dataset
WEEK_3/Superstore-SQL-Analysis/data/Sample - Superstore.csv - SQL stages · Table creation, load, subqueries, CTEs, window functions, final analysis
WEEK_3/Superstore-SQL-Analysis/sql/01_create_tables.sql - Results · Screenshots and narrative outputs for the SQL analysis
WEEK_3/Superstore-SQL-Analysis/screenshots
Week 4
Azure Data Factory pipeline
- Blob source · Azure Blob input dataset
WEEK_4/Azure-ADF-Data-Pipeline/Dataset/Sample-Superstore.csv - ADF pipeline · Get metadata and copy orchestration in Azure Data Factory
WEEK_4/Azure-ADF-Data-Pipeline/README.md - Blob sink · Destination blob storage target
WEEK_4/Azure-ADF-Data-Pipeline/Screenshots
Week 5
PySpark batch job
Week 6
Spark dual-format output
- Source CSV · Input file for Spark processing
WEEK_6/Spark_Assignment/data/source.csv - Spark flow · Transforms data and writes multiple output formats
WEEK_6/Spark_Assignment/main.py - CSV output · Spark output written as CSV
WEEK_6/Spark_Assignment/output/csv_output - Parquet output · Spark output written as Parquet
WEEK_6/Spark_Assignment/output/parquet_data
Week 7
Delta Lake notebook assignment
- Store CSV · Notebook input dataset
WEEK_7/DeltaLakeAssignment/data/sample-store.csv - Delta notebook · Notebook-based cleaning and transformation
WEEK_7/DeltaLakeAssignment/notebook/DeltaLakeAssignment.ipynb - Cleaned CSV · Processed store dataset written out
WEEK_7/DeltaLakeAssignment/output/cleaned_store.csv
Connections 14
- Raw CSV → Cleaning notebook: read · Notebook loads raw CSV for inspection and cleaning
- Cleaning notebook → Cleaned CSV: write · Cleansed dataset persisted to CSV
- Query script → Insights: derive · Query results summarized as insights and evidence
- Superstore CSV → SQL stages: load · Superstore data is staged into SQL analysis steps
- SQL stages → Results: document · Final analysis is captured in screenshots and outputs
- Blob source → ADF pipeline: ingest · ADF reads from the source blob
- ADF pipeline → Blob sink: copy · Pipeline moves data to destination blob
- Sales CSV → Spark job: process · PySpark job reads and transforms sales data
- Spark job → QA docs: evidence · Supporting assignment documentation accompanies the job
- Source CSV → Spark flow: process · Spark job consumes the source CSV
- Spark flow → CSV output: write · Transformed data persisted as CSV
- Spark flow → Parquet output: write · Transformed data persisted as Parquet
- Store CSV → Delta notebook: read · Notebook ingests the store CSV
- Delta notebook → Cleaned CSV: write · Cleaned output written back to disk