How melloddy/sparsechem works
Project type and purpose
• This is an open-source Python library (“SparseChem”) delivering fast, accurate machine learning models for biochemical datasets with extremely high-dimensional sparse inputs.
• It provides both an API (core modules under sparsechem/) and example scripts (examples/chembl/) to train, retrain, predict, profile memory, and fuse catalogues.
File structure
• Top-level:
• docs/ – Markdown documentation covering main usage, profiling, local retraining, catalogue fusion
• examples/chembl/ – pytest configuration and Python scripts demonstrating training, prediction, retraining workflows
• sparsechem/ – core package modules:
– data.py: data reading, sparse feature handling
– folding.py: fold generation and cross-validation splits
– models.py: model definition, architecture layers
– train.py: training loop, mixed-precision, GPU management
– predict.py: inference logic
– calculation_ProbCalibrationError.py: calibration error metrics
– utils.py, version.py: helpers and versioning
– test_*.py: unit tests
• setup.py – packaging and dependency declaration
• README.md, LICENSE, .gitignore
README for additional insights
• Emphasizes high-dimensional sparse inputs (millions of features/compounds).
• Points to four docs pages for general use, retraining, memory profiling, and catalogue fusion.
• Intended audience: ML engineers in cheminformatics.
Components
a. Main components
• User interface layer:
• example scripts or CLI wrapper
• user-provided sparse feature files (e.g., bit vectors, fingerprints)
• Core library modules:
• Data ingestion & preprocessing (sparsechem/data.py)
• Folding/cross-validation generator (sparsechem/folding.py)
• Model definitions (sparsechem/models.py)
• Training engine (sparsechem/train.py)
• Prediction engine (sparsechem/predict.py)
• Calibration error computation (sparsechem/calculation_ProbCalibrationError.py)
• Utilities (sparsechem/utils.py)
• External dependencies:
• PyTorch or TensorFlow (GPU, mixed precision)
• NumPy, SciPy (sparse matrix support)
• Artifacts & outputs:
• Trained model files (serialized weights)
• Calibration reports, logs, memory profile reports
• Predictions (CSV or database)
• Documentation & examples:
• docs/
• examples/chembl/
b. Relationships and interactions
• Data flow: user data file → Data ingestion → Preprocessing (sparse→dense conversion or direct sparse ops) → Folding → Model → Training engine ↔ GPU backend
• After training: model ↔ Calibration module → evaluation metrics
• Prediction flow: new data → Prediction engine → output CSV or downstream system
• Examples/scripts orchestrate these modules via the Python API
c. Architectural patterns/design principles
• Modular separation: data, modeling, training, prediction, calibration
• Plugin-style extensibility: utils and separate fold generator allow swapping components
• Clear API boundary between core library and example CLI scripts
• Test-driven: unit tests alongside each core module
d. Key technologies/frameworks
• Python packaging (setup.py)
• Sparse matrix libraries (SciPy.sparse)
• Deep learning framework (PyTorch or TensorFlow) for GPU training and mixed precision
• pytest for example tests (conftest.py)
• Markdown documentation