# klab-ai3/ml-polymer-recycling architecture

> Architecture diagram of the GitHub repository klab-ai3/ml-polymer-recycling, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/klab-ai3/ml-polymer-recycling
- Repository: https://github.com/klab-ai3/ml-polymer-recycling

## Overview

##### Project Type and Purpose
- This is a research-grade, end-to-end machine-learning pipeline for classifying polymer aging from Raman spectral data.
- It supports data ingestion, preprocessing, training (Figure2CNN & ResNet1D), reproducible validation, model artifact management, and inference via CLI and web dashboard.

##### High-Level Components
a. Data Layer
- datasets/rdwp (raw Raman spectra files)
b. Preprocessing & Training Pipeline
- scripts/discover_raman_files.py, preprocess_dataset.py, train_model.py
- validate_pipeline.sh orchestrates 10-fold CV, diagnostics, plotting
c. Model Definitions
- models/figure2_cnn.py, models/resnet_cnn.py
d. Model Artifacts & Logs
- outputs/ (model .pth files, JSON diagnostics, plots)
e. Inference Service
- backend/inference_utils.py (loading model, preprocessing input, running forward pass)
- backend/main.py (exposes inference API or CLI backend)
f. User Interfaces
- CLI tools (scripts/train_model.py, scripts/run_inference.py)
- Web Dashboard (dashboard/app.py)
- UI Application (app/ui_app.py)—future multi-modal UI wrapper
g. Configuration & Deployment
- environment.yml, HPC environment_hpc.yml
- docs/ (ENVIRONMENT_GUIDE, HPC_REMOTE_SETUP, REPRODUCIBILITY, BACKEND_MIGRATION_LOG)

##### Relationships & Data Flows
- Raw data → Preprocessing scripts → Prepared datasets
- Prepared datasets → Training script → Trained models → outputs/
- Validation script reads outputs/ + logs → Produces diagnostics JSON + plots
- Inference request (CLI or UI) invokes backend/main → uses inference_utils to load model from outputs → returns prediction → CLI or UI displays results
- Dashboard/app communicates with backend API to pull model registry (scans outputs/, JSON) and accept new inputs for batch inference

##### Architectural Patterns & Principles
- Modular ML pipeline (separate preprocessing, training, validation, inference phases)
- Clear separation of concerns: data layer, compute layer, service layer, presentation layer
- Reproducibility-first: versioned environment files, validate_pipeline.sh
- Extensible registry pattern: outputs folder as model registry that dashboard auto-discovers
- Branch strategy for local vs. HPC (main vs. hpc-main)

##### Key Technologies & Frameworks
- Core language: Python 3.10+
- Deep learning: PyTorch (CPU & CUDA)
- Data processing: NumPy, SciPy, Pandas, Scikit-learn
- Visualization: Matplotlib, Seaborn
- CLI parsing: ArgParse
- Web dashboard (likely Flask or FastAPI via dashboard/app.py)
- Deployment: Conda, cluster scripts (HPC_REMOTE_SETUP), shell script orchestration

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Layer
    subgraph "Data Layer"
        DL["datasets/"]:::data
    end

    %% Compute Layer
    subgraph "Compute & Training Pipeline"
        PR["discover_raman_files.py"]:::compute
        PP["preprocess_dataset.py"]:::compute
        TR["train_model.py"]:::compute
        VD["validate_pipeline.sh"]:::compute
        subgraph "Model Definitions"
            M1["figure2_cnn.py"]:::compute
            M2["resnet_cnn.py"]:::compute
        end
    end

    %% Model Artifacts
    OT["outputs/"]:::data

    %% Inference Service
    subgraph "Inference Service"
        IU["inference_utils.py"]:::service
        ME["main.py"]:::service
    end

    %% Presentation Layer
    subgraph "CLI Tools"
        CT1["scripts/train_model.py"]:::ui
        CT2["scripts/run_inference.py"]:::ui
    end
    subgraph "Web & UI Applications"
        WD["dashboard/app.py"]:::ui
        UI["app/ui_app.py"]:::ui
    end

    %% Configuration & Documentation
    subgraph "Configuration & Docs"
        C1["ENVIRONMENT_GUIDE.md"]:::config
        C2["HPC_REMOTE_SETUP.md"]:::config
        C3["REPRODUCIBILITY.md"]:::config
        C4["BACKEND_MIGRATION_LOG.md"]:::config
    end

    %% Data Flow
    DL --> PR --> PP --> TR --> OT
    TR --> VD --> OT
    M1 --> TR
    M2 --> TR

    %% Inference Flow
    OT --> IU --> ME
    CT1 --> TR
    CT2 --> ME
    WD --> ME
    UI --> ME

    %% Docs Link
    VD --> C3

    %% Click Events
    click DL "https://github.com/klab-ai3/ml-polymer-recycling/tree/main/datasets/"
    click PR "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/scripts/discover_raman_files.py"
    click PP "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/scripts/preprocess_dataset.py"
    click TR "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/scripts/train_model.py"
    click VD "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/validate_pipeline.sh"
    click M1 "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/models/figure2_cnn.py"
    click M2 "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/models/resnet_cnn.py"
    click OT "https://github.com/klab-ai3/ml-polymer-recycling/tree/main/outputs/"
    click IU "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/backend/inference_utils.py"
    click ME "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/backend/main.py"
    click CT1 "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/scripts/train_model.py"
    click CT2 "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/scripts/run_inference.py"
    click WD "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/dashboard/app.py"
    click UI "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/app/ui_app.py"
    click C1 "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/docs/ENVIRONMENT_GUIDE.md"
    click C2 "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/docs/HPC_REMOTE_SETUP.md"
    click C3 "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/docs/REPRODUCIBILITY.md"
    click C4 "https://github.com/klab-ai3/ml-polymer-recycling/blob/main/docs/BACKEND_MIGRATION_LOG.md"

    %% Styles
    classDef data fill:#AED6F1,stroke:#1B4F72,color:#1B2631;
    classDef compute fill:#A9DFBF,stroke:#1D8348,color:#145A32;
    classDef service fill:#F5B041,stroke:#B9770E,color:#7E5109;
    classDef ui fill:#D7BDE2,stroke:#6C3483,color:#512E5F;
    classDef config fill:#D5D8DC,stroke:#566573,color:#212F3C
```
