# melloddy/sparsechem architecture

> Architecture diagram of the GitHub repository melloddy/sparsechem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/melloddy/sparsechem
- Repository: https://github.com/melloddy/sparsechem

## Overview

##### Project type and purpose
- This is an open-source Python library (“SparseChem”) delivering fast, accurate machine learning models for biochemical datasets with extremely high-dimensional sparse inputs.
- It provides both an API (core modules under sparsechem/) and example scripts (examples/chembl/) to train, retrain, predict, profile memory, and fuse catalogues.

##### File structure
- Top-level:
- docs/ – Markdown documentation covering main usage, profiling, local retraining, catalogue fusion
- examples/chembl/ – pytest configuration and Python scripts demonstrating training, prediction, retraining workflows
- sparsechem/ – core package modules:
– data.py: data reading, sparse feature handling
– folding.py: fold generation and cross-validation splits
– models.py: model definition, architecture layers
– train.py: training loop, mixed-precision, GPU management
– predict.py: inference logic
– calculation_ProbCalibrationError.py: calibration error metrics
– utils.py, version.py: helpers and versioning
– test_*.py: unit tests
- setup.py – packaging and dependency declaration
- README.md, LICENSE, .gitignore

##### README for additional insights
- Emphasizes high-dimensional sparse inputs (millions of features/compounds).
- Points to four docs pages for general use, retraining, memory profiling, and catalogue fusion.
- Intended audience: ML engineers in cheminformatics.

##### Components
a. Main components
- User interface layer:
- example scripts or CLI wrapper
- user-provided sparse feature files (e.g., bit vectors, fingerprints)
- Core library modules:
- Data ingestion & preprocessing (sparsechem/data.py)
- Folding/cross-validation generator (sparsechem/folding.py)
- Model definitions (sparsechem/models.py)
- Training engine (sparsechem/train.py)
- Prediction engine (sparsechem/predict.py)
- Calibration error computation (sparsechem/calculation_ProbCalibrationError.py)
- Utilities (sparsechem/utils.py)
- External dependencies:
- PyTorch or TensorFlow (GPU, mixed precision)
- NumPy, SciPy (sparse matrix support)
- Artifacts & outputs:
- Trained model files (serialized weights)
- Calibration reports, logs, memory profile reports
- Predictions (CSV or database)
- Documentation & examples:
- docs/
- examples/chembl/

b. Relationships and interactions
- Data flow: user data file → Data ingestion → Preprocessing (sparse→dense conversion or direct sparse ops) → Folding → Model → Training engine ↔ GPU backend
- After training: model ↔ Calibration module → evaluation metrics
- Prediction flow: new data → Prediction engine → output CSV or downstream system
- Examples/scripts orchestrate these modules via the Python API

c. Architectural patterns/design principles
- Modular separation: data, modeling, training, prediction, calibration
- Plugin-style extensibility: utils and separate fold generator allow swapping components
- Clear API boundary between core library and example CLI scripts
- Test-driven: unit tests alongside each core module

d. Key technologies/frameworks
- Python packaging (setup.py)
- Sparse matrix libraries (SciPy.sparse)
- Deep learning framework (PyTorch or TensorFlow) for GPU training and mixed precision
- pytest for example tests (conftest.py)
- Markdown documentation

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User & Examples
    subgraph "User Interface & Examples"
        UI["User Data Files"]:::user
        NewInput["New Input Data"]:::user
        EX_train["Train Script (examples/chembl/train.py)"]:::example
        EX_retrain["Retrain Script (examples/chembl/retrain.py)"]:::example
        EX_predict["Predict Script (examples/chembl/predict.py)"]:::example
        EX_predict_local["Predict Local Trunk (examples/chembl/predict_local_trunk.py)"]:::example
        EX_conftest["pytest Config (examples/chembl/conftest.py)"]:::example
        EX_test_train["Example Tests (examples/chembl/test_train.py)"]:::example
    end

    %% Core Library
    subgraph "Core Library" 
        DI["Data Ingestion & Preprocessing<br/>(sparsechem/data.py)"]:::core
        FG["Folding / Cross-Validation<br/>(sparsechem/folding.py)"]:::core
        MD["Model Definitions & Architecture<br/>(sparsechem/models.py)"]:::core
        TE["Training Engine<br/>(sparsechem/train.py)"]:::core
        PE["Prediction Engine<br/>(sparsechem/predict.py)"]:::core
        CE["Calibration Error Computation<br/>(sparsechem/calculation_ProbCalibrationError.py)"]:::core
        UT["Utilities & Helpers<br/>(sparsechem/utils.py)"]:::core
        VR["Versioning Module<br/>(sparsechem/version.py)"]:::core
        INIT["Core Init<br/>(sparsechem/__init__.py)"]:::core
        TEST_cal["Test Calibration<br/>(sparsechem/test_calibration.py)"]:::test
        TEST_loss["Test Losses<br/>(sparsechem/test_losses.py)"]:::test
        TEST_sparse["Test Sparse Data<br/>(sparsechem/test_sparse.py)"]:::test
    end

    %% External Dependencies
    subgraph "External Dependencies"
        Torch["PyTorch / TensorFlow"]:::external
        SciPy["NumPy / SciPy.sparse"]:::external
    end

    %% Artifacts & Outputs
    subgraph "Artifacts & Outputs"
        MA[(Trained Model Files)]:::artifact
        EM[(Evaluation Metrics & Reports)]:::artifact
        PO[(Predictions Output)]:::artifact
    end

    %% Documentation & Packaging
    subgraph "Documentation & Packaging"
        DOC_main["Usage Guide<br/>(docs/main.md)"]:::doc
        DOC_local["Local Retraining Guide<br/>(docs/local_trunk.md)"]:::doc
        DOC_mem["Memory Profiling Guide<br/>(docs/memprofile.md)"]:::doc
        DOC_fusion["Catalogue Fusion Guide<br/>(docs/catalogue_fusion.md)"]:::doc
        SETUP["Packaging & Metadata<br/>(setup.py)"]:::meta
        README["README.md"]:::meta
        LICENSE["LICENSE"]:::meta
        GITIGNORE[".gitignore"]:::meta
    end

    %% Data Flow
    UI -->|"load data"| DI -->|"generate folds"| FG -->|"train calls"| TE -->|"produce artifact"| MA
    MA -->|"compute calibration"| CE -->|"generate metrics"| EM
    NewInput -->|"infer"| PE -->|"write output"| PO

    %% Core interactions
    DI -->|"provide features"| MD
    TE -->|"use model defs"| MD
    TE -->|"GPU & mixed precision"| Torch
    TE -->|"sparse ops"| SciPy
    PE -->|"load model"| MA
    CE -->|"use metrics"| UT

    %% Example orchestration
    EX_train --> DI
    EX_train --> FG
    EX_train --> TE
    EX_retrain --> TE
    EX_predict --> PE
    EX_predict_local --> PE
    EX_conftest --> EX_test_train

    %% Click Events
    click DI "https://github.com/melloddy/sparsechem/blob/master/sparsechem/data.py"
    click FG "https://github.com/melloddy/sparsechem/blob/master/sparsechem/folding.py"
    click MD "https://github.com/melloddy/sparsechem/blob/master/sparsechem/models.py"
    click TE "https://github.com/melloddy/sparsechem/blob/master/sparsechem/train.py"
    click PE "https://github.com/melloddy/sparsechem/blob/master/sparsechem/predict.py"
    click CE "https://github.com/melloddy/sparsechem/blob/master/sparsechem/calculation_ProbCalibrationError.py"
    click UT "https://github.com/melloddy/sparsechem/blob/master/sparsechem/utils.py"
    click VR "https://github.com/melloddy/sparsechem/blob/master/sparsechem/version.py"
    click INIT "https://github.com/melloddy/sparsechem/blob/master/sparsechem/__init__.py"
    click TEST_cal "https://github.com/melloddy/sparsechem/blob/master/sparsechem/test_calibration.py"
    click TEST_loss "https://github.com/melloddy/sparsechem/blob/master/sparsechem/test_losses.py"
    click TEST_sparse "https://github.com/melloddy/sparsechem/blob/master/sparsechem/test_sparse.py"
    click EX_train "https://github.com/melloddy/sparsechem/blob/master/examples/chembl/train.py"
    click EX_retrain "https://github.com/melloddy/sparsechem/blob/master/examples/chembl/retrain.py"
    click EX_predict "https://github.com/melloddy/sparsechem/blob/master/examples/chembl/predict.py"
    click EX_predict_local "https://github.com/melloddy/sparsechem/blob/master/examples/chembl/predict_local_trunk.py"
    click EX_conftest "https://github.com/melloddy/sparsechem/blob/master/examples/chembl/conftest.py"
    click EX_test_train "https://github.com/melloddy/sparsechem/blob/master/examples/chembl/test_train.py"
    click DOC_main "https://github.com/melloddy/sparsechem/blob/master/docs/main.md"
    click DOC_local "https://github.com/melloddy/sparsechem/blob/master/docs/local_trunk.md"
    click DOC_mem "https://github.com/melloddy/sparsechem/blob/master/docs/memprofile.md"
    click DOC_fusion "https://github.com/melloddy/sparsechem/blob/master/docs/catalogue_fusion.md"
    click SETUP "https://github.com/melloddy/sparsechem/blob/master/setup.py"
    click README "https://github.com/melloddy/sparsechem/blob/master/README.md"
    click LICENSE "https://github.com/melloddy/sparsechem/tree/master/LICENSE"
    click GITIGNORE "https://github.com/melloddy/sparsechem/blob/master/.gitignore"

    %% Styles
    classDef core fill:#AED6F1,stroke:#1B4F72,color:#1B2631
    classDef example fill:#ABEBC6,stroke:#196F3D,color:#0B5345
    classDef external fill:#F9E79F,stroke:#B7950B,color:#784212
    classDef artifact fill:#D5D8DC,stroke:#566573,color:#1B2631
    classDef doc fill:#F5B7B1,stroke:#943126,color:#641E16
    classDef meta fill:#D6EAF8,stroke:#1A5276,color:#154360
    classDef user fill:#FAD7A0,stroke:#AF601A,color:#6E2C00
    classDef test fill:#E8DAEF,stroke:#6C3483,color:#4A235A
```
