How giabach1106/chest_x_ray_abnormal works
Project type and purpose
• This is a domain‐specific machine learning pipeline for detecting abnormalities in chest X-ray images (Kaggle “vinbigdata” competition).
• It comprises multiple ML sub-pipelines: data preparation, classification (EfficientNetB4), object detection (YOLOv4 via Darknet, YOLOv5 & YOLOv7 via PyTorch), evaluation, post-processing, and submission generation.
File structure
• Top‐level folders:
• DATA (raw CSVs and kept placeholders)
• src/prepare_data (data cleaning, augmentation, splits)
• src/classification/EfficientNetB4 (training/eval scripts & notebooks, WandB logs)
• src/detection/yolov4/darknet (C/C++ build & inference for YOLOv4)
• src/detection/yolov5 and src/detection/yolov7 (PyTorch‐based detection pipelines, configs, utils, Flask/Triton deploy examples)
• src/eval, src/post_processing (notebooks for metric calculation and result refinement)
• Notice build scripts (Makefile, CMakeLists.txt, Dockerfiles), shell wrappers (.sh/.ps1), Jupyter notebooks for experimentation, and WandB/Comet tracking folders.
Key insights from README
• Focus: lung anomaly detection via classification + localization/detection.
• No centralized web front end—pure ML research code.
• Evaluation and submission CSVs are generated as final deliverables.
Main components & their interactions
a. Data Storage
• Raw images + annotation CSVs in DATA/ and external storage (e.g., S3 or local disk).
b. Data Preparation
• src/prepare_data: processes raw CSVs and images → augmented datasets, train/val/test splits.
→ Output: ready-to-consume datasets for each model.
c. Classification Pipeline
• src/classification/EfficientNetB4
– Python scripts & notebooks train EfficientNetB4 on binary labels.
– Uses WandB for experiment tracking.
– Produces classification model and CSV predictions.
d. Detection Pipelines
• YOLOv4 (Darknet C/C++)
– Build via Make/CMake → darknet executable
– Scripts for training, inference, streaming
• YOLOv5 & YOLOv7 (PyTorch)
– python train.py / detect.py
– YAML config files define architectures/datasets
– Utils: custom dataloaders, augmentations, experiment logging (WandB, Comet, ClearML), Dockerfiles for deployment, example Flask/Triton server
– Output: PyTorch models (.pt), ONNX/TensorRT artifacts, CSV submissions
e. Evaluation & Post-Processing
• src/eval: Jupyter notebooks compute metrics (mAP, accuracy)
• src/post_processing: refine detection masks/predictions
f. Submission
• CSV writers in each pipeline produce final submission files for competition.
Architectural patterns & design principles
• Modular ML pipelines: separate directories per model and stage
• Experiment reproducibility: notebooks + shell wrappers + environment files (.yaml, requirements.txt)
• Extensibility: multiple detection back-ends can be swapped (Darknet vs PyTorch)
• Clear data flow: raw → processed → train/test → model → eval → submission
• Infrastructure as code: Dockerfiles and CI config in detection folders
Technologies & frameworks
• Python (ML logic, PyTorch, Flask/Triton)
• Jupyter Notebooks (experimental workflows)
• Bash/PowerShell scripts (automation)
• Darknet (C/C++, GPU kernels)
• WandB/Comet/ClearML (tracking)
• Docker (containerization)
• YAML (configs for models & datasets)