How shapniljoy/eda--and--inferential--analysis works
1. Repository purpose
- This is a data analysis repo, not an application or service.
- Its job is to turn the Global Cancer Patients 2015–2024 Kaggle dataset into an exploratory report with statistical findings, visualizations, and a small predictive modeling component.
2. Primary artifacts
- eda-and-inferential-analysis.ipynb is the main working notebook and likely contains the full analysis workflow.
- EDA Report.pdf is the published/exported report.
- README.md is the narrative summary of the analysis and findings.
3. Core data flow
- The workflow starts with a patient-level dataset containing demographics, risk factors, cancer characteristics, costs, survival, and a target severity score.
- Data is then explored univariately, tested statistically in pairwise/multivariate views, and finally fed into a Random Forest feature-importance analysis.
- Outputs are visualizations, statistical test results, model scores, and the final report narrative.
4. Analytical pipeline boundary
- The repo is organized around a single notebook-driven pipeline rather than separated modules or services.
- The main boundary is between:
- raw dataset / source data
- analysis notebook
- reporting artifacts (README.md, PDF)
- There is no evidence of a database, API, UI, or scheduled job layer.
5. Descriptive statistics subsystem
- The first analysis phase covers distributions of age, gender, country, risk factors, cancer type, stage, treatment cost, and severity.
- This subsystem is responsible for histograms, KDE plots, boxplots, Q-Q plots, bar charts, and pie charts.
- Its output is baseline characterization of the dataset and outlier/normality assessment.
6. Inferential statistics subsystem
- The second phase evaluates relationships using Pearson/Spearman correlations, t-tests, one-way ANOVA, chi-square tests, and OLS regression.
- This part of the notebook answers questions like:
- whether risk factors relate to severity
- whether treatment cost differs across countries or genders
- whether stage and cancer type are associated
- These tests form the statistical backbone of the report.
7. Predictive modeling subsystem
- A Random Forest Regressor is used as the only explicit machine-learning model.
- It is trained with an 80/20 split and used for two targets:
- Target_Severity_Score
- Survival_Years
- Feature importance is used as a ranking mechanism to identify the strongest predictors of severity.
8. Feature engineering / variable handling
- The notebook distinguishes between:
- numerical standardized risk factors such as genetic risk, air pollution, alcohol use, smoking, obesity
- categorical variables such as gender, country, year, cancer type, and stage
- It also constructs at least one interaction term: Genetic_Risk × Smoking, plus a transformed version for regression testing.
- This is the main modeling boundary where raw fields are turned into analytical features.
9. Outcome-focused analytical themes
- The report is organized around seven thematic questions:
- distributions of demographics and clinical variables
- risk factors vs severity
- early-stage detection by cancer type
- predictors of severity and survival
- treatment cost burden
- stage effects on cost and survival
- genetic risk × smoking interaction
- These themes define the repository’s logical structure more than code modules do.
10. Visualization and reporting layer
- Visual output is generated with Seaborn, Matplotlib, and Plotly.
- The notebook likely serves both as computation and presentation, with charts embedded inline.
- The PDF is the final consumable artifact for sharing findings outside the notebook environment.
11. Key architectural constraint
- The dataset appears to be prestructured and standardized: several risk factors share the same scale and distribution shape.
- Because of that, the notebook is focused on analysis and interpretation, not data ingestion, cleansing pipelines, or heavy transformation logic.
12. Practical diagram takeaway
- If you were drawing this repo, it would be best represented as:
- one source dataset entering a notebook-centric analysis pipeline,
- branching into descriptive stats, inferential tests, and Random Forest modeling,
- converging into written conclusions and exported report artifacts.
- The repo is essentially a single analytical workflow with reporting outputs, not a multi-component system.
Main entry points: eda-and-inferential-analysis.ipynb Main notebook · README.md README · EDA Report.pdf PDF report
Components 16
Source Data
Patient-level cancer dataset and derived analytical fields
- Cancer dataset · Global Cancer Patients 2015–2024 patient-level dataset
Notebook Pipeline
Exploratory analysis, statistical testing, and modeling in the main notebook
- Main notebook · Primary workflow for exploration, tests, modeling, and visualization
eda-and-inferential-analysis.ipynb - Feature setup · Standardized numeric risks, categorical fields, and interaction terms
- Descriptive stats · Distributions, outliers, normality checks, and baseline summaries
- Distribution plots · Histograms, KDEs, boxplots, Q-Q plots, bar charts, and pie charts
- Inferential tests · Correlation, t-test, ANOVA, chi-square, and OLS analysis
- Risk vs severity · Tests how risk factors relate to target severity
- Group effects · Compares cost, survival, and severity across categories
- Association tests · Checks stage, cancer type, and other categorical relationships
- Risk interaction · Genetic risk × smoking interaction and transformed variant
- Random forest · Random Forest regressor used for severity and survival prediction
- Feature importance · Ranks predictors of severity from the fitted forest model
- Model scores · 80/20 split evaluation for severity and survival targets
Reporting
Narrative and exported deliverables
- Plots and tables · Inline figures, statistical results, and model summaries
- README · Written summary of findings and interpretation
README.md - PDF report · Exported report for sharing outside the notebook
EDA Report.pdf
Connections 16
- Cancer dataset → Main notebook: analyze · Notebook consumes the structured patient dataset
- Main notebook → Feature setup: prepare · Numeric, categorical, and interaction features are defined
- Feature setup → Descriptive stats: branch · Prepared variables feed exploratory summaries
- Descriptive stats → Distribution plots: render · Summaries are visualized as distribution charts
- Feature setup → Inferential tests: test · Engineered variables support statistical testing
- Inferential tests → Risk vs severity: answer · Evaluates relationships between risk factors and severity
- Inferential tests → Group effects: answer · Compares outcomes across countries, genders, and stages
- Inferential tests → Association tests: answer · Tests categorical associations such as stage and cancer type
- Feature setup → Risk interaction: derive · Creates the genetic risk and smoking interaction feature
- Feature setup → Random forest: fit · Model uses the prepared analytical feature set
- Risk interaction → Random forest: include · Interaction feature is available to the model and regression tests
- Random forest → Model scores: evaluate · Produces train-test scores for severity and survival targets
- Random forest → Feature importance: rank · Outputs predictor importance for severity interpretation
- Main notebook → Plots and tables: emit · Notebook generates figures, tables, and statistical summaries
- Plots and tables → README: summarize · Key findings are written into the repository summary
- Plots and tables → PDF report: publish · Final report is exported as a PDF artifact