How otniel113/lungcanceridentification works
Project Type and Purpose
• This is a data-science/ML pipeline for “Identification of Lung Cancer in Smoker Person Using Ensemble Methods Based on Gene Expression Data.”
• It is not a web or full-stack app; instead it’s a sequence of Jupyter-based ETL, feature-engineering, training, hyperparameter tuning, validation, and inference steps.
Main Architectural Components
a. Data Sources
• Raw DNA microarray gene-expression files (not shown in file tree but implied)
• Sample labels/metadata
b. Preprocessing Stage
• Notebook “01_PersiapanData.ipynb”
• Cleans, normalizes, and merges raw data → outputs preprocessed tables
c. Feature Engineering Stage
• Notebook “02_Praproses_Fitur.ipynb”
• Selects and possibly reduces features → writes feature sets to features/*.pkl
d. Hyperparameter Tuning & Model Training
• “03A_HyperTuning_RF.ipynb” (Random Forest)
• “03B_HyperTuning_AB.ipynb” (AdaBoost)
• Each explores parameter grid via scikit-learn, outputs best estimators to models/*.pkl
e. Validation & Inference
• “04A_ValidasiModel_RF.ipynb” & “04B_ValidasiModel_AB.ipynb”
• Load each model, run predictions on test data → writes hasil_prediksi/*.csv
f. Artifacts & Storage
• features/ (feature pickles)
• models/ (trained model pickles)
• hasil_prediksi/ (prediction CSVs)
Relationships & Data Flow
• Raw Data → Preprocessing Notebook → Preprocessed Data
• Preprocessed Data → Feature Engineering Notebook → feature‐store (features/*.pkl)
• feature‐store + hyperparameter scripts → trained-model store (models/*.pkl)
• trained models + test data → validation notebooks → prediction results (CSV)
Architectural Patterns & Principles
• ETL pipeline: sequential notebook stages that each read from/write to disk.
• Modular separation: each logical step (prep, feature, train, validate) is in its own notebook.
• Artifact versioning: distinct directories for features, models, predictions.
• Reproducibility: all steps are Jupyter notebooks using Python + scikit-learn.