How ebrahim-tambuwala/sofnexis_data_analysis_with_python works
What this project is (type + purpose)
This repository is not a deployed “system” in the traditional sense; it’s a data analysis / data science notebook workspace for an internship. The primary “runtime” is a local (or hosted) Jupyter environment executing Python notebooks against CSV datasets, producing cleaned CSV outputs and analytical results (EDA, time-series analysis, hypothesis testing, ML modeling).
How the file tree maps to architecture
Top-level structure:
• Task_1/ … Task_5/: five independent analysis modules (each is effectively a “component”).
• Each task is centered around one or more .ipynb notebooks and CSV datasets.
• Task 1 is special: it contains three sub-datasets (Customers, Organizations, People) and produces cleaned CSV outputs.
Key artifacts:
• Inputs:
• Task_1/*/customers-100.csv, organizations-100.csv, people-100.csv
• Task_2/titanic.csv
• Task_3/airline-passengers.csv
• Task_4/ab_data.csv
• Task_5/titanic.csv
• Processing:
• Jupyter notebooks (Task*_*.ipynb) contain the transformation/analysis logic.
• Outputs:
• Task_1/*/cleaned_*_data.csv are explicit persisted outputs.
• Other tasks likely generate plots, metrics, model results, but they are not persisted in the repo (so represent them as “in-notebook outputs” unless you confirm they write files).
Actor / Operator
• “Analyst / Intern” running notebooks.
Execution Environment
• “Jupyter Notebook (Python)” as the primary runtime.
Data Sources (CSV files)
• Task 1 raw CSVs (customers/org/people)
• Titanic dataset (used in Task 2 + Task 5; show it reused)
• Airline passengers dataset (Task 3)
• A/B dataset (Task 4)
Processing Modules (Notebooks)
• Notebook as a component:
• Task1_Data_Cleaning_Customers.ipynb
• Task1_Data_Cleaning_Organization.ipynb
• Task1_Data_Cleaning_People.ipynb
• Task2_EDA.ipynb
• Task3_Time_Series_Analysis.ipynb
• Task4_Hypothesis_Testing.ipynb
• Task_5_Machine_Learning_Modeling.ipynb
Outputs / Artifacts
• Explicit persisted outputs (Task 1 cleaned CSVs) should be drawn as artifact storage:
• cleaned_customers_data.csv
• cleaned_organizations_data.csv
• cleaned_people_data.csv
• “EDA Visualizations + Summary Stats”
• “Time-series plots + forecast results (if any)”
• “Hypothesis test metrics (p-values, confidence intervals)”
• “Trained model + evaluation metrics” (even if not saved, it’s still an output of the notebook execution)
Task 1 (Data Cleaning pipeline)
• customers-100.csv → Task1_Data_Cleaning_Customers.ipynb → cleaned_customers_data.csv
• organizations-100.csv → Task1_Data_Cleaning_Organization.ipynb → cleaned_organizations_data.csv
• people-100.csv → Task1_Data_Cleaning_People.ipynb → cleaned_people_data.csv
Task 2 (EDA)
• titanic.csv → Task2_EDA.ipynb → “EDA outputs (charts/tables)”
Task 3 (Time Series)
• airline-passengers.csv → Task3_Time_Series_Analysis.ipynb → “Time-series outputs”
Task 4 (Hypothesis Testing)
• ab_data.csv → Task4_Hypothesis_Testing.ipynb → “Test results (p-value, effect size, CI)”
Task 5 (ML Modeling)
• titanic.csv → Task_5_Machine_Learning_Modeling.ipynb → “Model + metrics”
• Important: show titanic.csv as a shared input reused by Task 2 and Task 5.
Orchestration
• “Analyst” → triggers execution of each notebook in Jupyter.
• Jupyter → reads/writes CSVs on local filesystem (this repo).
Architectural patterns present (and how to reflect them)
This repo follows a task-oriented, notebook-per-problem pattern:
• Each Task_* directory is effectively a self-contained analysis module.
• Inputs are local flat files (CSV), not databases or APIs.
• Outputs are either persisted files (Task 1) or ephemeral notebook results (Tasks 2–5).
• Inside each task boundary: show notebook + its dataset(s) + outputs.
What not to overcomplicate
• Don’t invent services (APIs, DBs, queues) that aren’t present.