How julian-borges/breast-cancer-ai-misclassification- works
Project type and purpose
• This is a data‐science/ML pipeline for auditing shortcut learning in breast cancer genomic subtyping. It is not a web service or full‐stack app but a research/analysis workflow that ingests raw clinical/genomic data, preprocesses it, trains & explains ML models, and outputs visualizations and reports.
File structure
• Top‐level files and their roles:
• clinical_raw.dta – Original raw clinical dataset in Stata format
• AI-BIAS cleaned_TCGA_PAM50_model_dataset.csv – Cleaned and combined dataset (gene expression + clinical features) for modeling
• PAM50_Capstone_Consolidated_Pipeline.do – Stata script for data cleaning/preprocessing steps (pseudo‐SHAP calculations, merging)
• PAM50_SHAP_Colab_Notebook.ipynb – Python notebook that trains XGBoost, runs true SHAP, and generates plots
• requirements.txt – Python library dependencies (e.g., xgboost, shap, pandas)
• README.md – Project overview, objectives, and instructions
Key insights from the README
• Pipeline stages:
a. Data cleaning (Stata)
b. Model training & validation (XGBoost in Python)
c. Explainability & auditing (SHAP)
d. Misclassification analysis by subgroup
• Execution environment: Google Colab for the notebook
Main components & interactions
a. Data Ingestion & Preprocessing
– Input: clinical_raw.dta
– Tool: Stata (.do script)
– Output: cleaned CSV (AI-BIAS cleaned_TCGA_PAM50_model_dataset.csv)
b. Model Training & Validation
– Input: cleaned CSV
– Tool: Python environment (Google Colab)
– Framework: XGBoost for classification (PAM50 subtypes)
– Output: trained model artifacts (in‐memory in notebook)
c. Explainability
– Input: trained model + dataset
– Library: SHAP (python package)
– Output: feature‐importance plots, global & local explanations
d. Misclassification Analysis
– Input: model predictions + actual subtypes + clinical covariates
– Tool: Python (pandas/matplotlib/seaborn)
– Output: analyses by age, hormone status, batch effects
e. Reporting & Visualization
– Notebook generates figures and tables for the capstone report
Architectural patterns & design principles
• Sequential pipeline (ETL → ML → explain)
• Separation of concerns: Stata handles raw data cleaning, Python handles modeling & explainability
• Reproducibility: version‐controlled scripts (.do, notebook) and requirements.txt
• Components:
• Rectangles for data stores (.dta, .csv)
• Directional data flow (e.g., raw→Stata→clean CSV→Python→model→explain→report)
• Blue for data artifacts
• Green for compute environments/tools
• Orange for outputs/visualizations
• Labels:
• File names (clinical_raw.dta, AI-BIAS cleaned_TCGA_PAM50_model_dataset.csv)
• Scripts (PAM50_Capstone_Consolidated_Pipeline.do, PAM50_SHAP_Colab_Notebook.ipynb)
• Libraries (XGBoost, SHAP)