How klab-ai3/ml-polymer-recycling works
Project Type and Purpose
• This is a research-grade, end-to-end machine-learning pipeline for classifying polymer aging from Raman spectral data.
• It supports data ingestion, preprocessing, training (Figure2CNN & ResNet1D), reproducible validation, model artifact management, and inference via CLI and web dashboard.
High-Level Components
a. Data Layer
• datasets/rdwp (raw Raman spectra files)
b. Preprocessing & Training Pipeline
• scripts/discover_raman_files.py, preprocess_dataset.py, train_model.py
• validate_pipeline.sh orchestrates 10-fold CV, diagnostics, plotting
c. Model Definitions
• models/figure2_cnn.py, models/resnet_cnn.py
d. Model Artifacts & Logs
• outputs/ (model .pth files, JSON diagnostics, plots)
e. Inference Service
• backend/inference_utils.py (loading model, preprocessing input, running forward pass)
• backend/main.py (exposes inference API or CLI backend)
f. User Interfaces
• CLI tools (scripts/train_model.py, scripts/run_inference.py)
• Web Dashboard (dashboard/app.py)
• UI Application (app/ui_app.py)—future multi-modal UI wrapper
g. Configuration & Deployment
• environment.yml, HPC environment_hpc.yml
• docs/ (ENVIRONMENT_GUIDE, HPC_REMOTE_SETUP, REPRODUCIBILITY, BACKEND_MIGRATION_LOG)
Relationships & Data Flows
• Raw data → Preprocessing scripts → Prepared datasets
• Prepared datasets → Training script → Trained models → outputs/
• Validation script reads outputs/ + logs → Produces diagnostics JSON + plots
• Inference request (CLI or UI) invokes backend/main → uses inference_utils to load model from outputs → returns prediction → CLI or UI displays results
• Dashboard/app communicates with backend API to pull model registry (scans outputs/, JSON) and accept new inputs for batch inference
Architectural Patterns & Principles
• Modular ML pipeline (separate preprocessing, training, validation, inference phases)
• Clear separation of concerns: data layer, compute layer, service layer, presentation layer
• Reproducibility-first: versioned environment files, validate_pipeline.sh
• Extensible registry pattern: outputs folder as model registry that dashboard auto-discovers
• Branch strategy for local vs. HPC (main vs. hpc-main)
Key Technologies & Frameworks
• Core language: Python 3.10+
• Deep learning: PyTorch (CPU & CUDA)
• Data processing: NumPy, SciPy, Pandas, Scikit-learn
• Visualization: Matplotlib, Seaborn
• CLI parsing: ArgParse
• Web dashboard (likely Flask or FastAPI via dashboard/app.py)
• Deployment: Conda, cluster scripts (HPC_REMOTE_SETUP), shell script orchestration