# rdkit/mmpdb architecture

> Architecture diagram of the GitHub repository rdkit/mmpdb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rdkit/mmpdb
- Repository: https://github.com/rdkit/mmpdb

## Overview

##### Project Type and Purpose
- This is a command‐line, open-source cheminformatics toolkit (“mmpdb”) written in Python.
- It implements a pipeline for (a) fragmenting molecular structures (SMILES), (b) indexing matched molecular pairs into a SQLite (or Postgres) database, and (c) analyzing or generating new molecules based on rule transformations and property predictions.

##### High-Level Components
a. User / CLI Layer
- The `mmpdb` executable, implemented with the Click library.
- Subcommands: fragment, index, loadprops, transform, predict, generate, plus help-* and distributed utilities (smi_split, fragdb_partition, fragdb_merge).
b. Core Library (`mmpdblib/`)
- CLI dispatch (`cli/…` modules)
- Fragmentation engines (`fragment_algorithm.py`, `analysis_algorithms.py`, `rgroup2smarts.py`, `smiles_syntax.py`)
- File I/O (`fileio.py`) – reading SMILES, writing fragment databases (.fragdb files).
- Database utilities (`dbutils.py`, `schema.py`, SQL scripts in `.sql` files).
- Indexing engines (`index_algorithm.py`, `index_types.py`, `index_writers.py`) using the Peewee ORM.
- Property I/O (`properties_io.py`) and reporting (`reporters.py`).
- Environment/fingerprint generation (`environment.py`, `cansmirks_table.py`).
c. Data Stores & Formats
- Flat‐file fragment database: `.fragdb` (binary or JSON‐like).
- SQLite (or Postgres) MMP database `.mmpdb` with tables: compounds, rule_smiles, rules, environments, rule_stats, pairs.
- Optional cache files for fragmentation.
d. External Dependencies
- RDKit for cheminformatics operations
- NumPy, SciPy for data and statistics
- Click for CLI
- Peewee (and optional psycopg2) for database access
- psutil for memory metrics in index

##### Interactions and Data Flow
1) User invokes `mmpdb fragment` → CLI module picks up SMILES file → calls fileio + fragment_algorithm → produces a `.fragdb` file (with optional caching).
2) User invokes `mmpdb index` → CLI module loads `.fragdb` → index_algorithm groups by constant fragments → writes SQLite `.mmpdb` via Peewee.
3) `mmpdb loadprops` → reads CSV properties → updates `.mmpdb` tables and recomputes rule statistics.
4) Query commands (`transform`, `predict`) → read `.mmpdb` → analysis_algorithms compute matched pair transformations and property shifts → output to console or file.
5) `mmpdb generate` → leverages rules + environments → produces new SMILES suggestions.
6) Distributed utilities (`smi_split`, `fragdb_partition`, `fragdb_merge`) operate on `.smi` or `.fragdb` shards in parallel, then merge results.

##### Architectural Patterns and Principles
- Modular CLI with subcommand pattern (Click).
- Separation of concerns: I/O, core algorithms, database access are isolated in separate modules.
- Data-driven pipeline: input files → intermediate flat files → relational DB → analysis.
- Use of caching and partitioning to scale large datasets.
- Lightweight ORM for portability (SQLite + optional Postgres).

- User / CLI Layer
- Core Library Modules
- Storage & Cache (flat-files, DB)
- External Services / Libraries
- “mmpdb CLI (Click)” at top.
- From SMILES input → Fragmenter → `.fragdb`
- From `.fragdb` → Indexer → SQLite DB
- From CSV Properties → Property Loader → SQLite DB
- From SQLite DB → Analyzer / Transformer / Generator → console or output file
- From Fragment & Index → optional partition/merge utilities → merged fragments → index
- Core algorithm modules in another (green).
- External libraries in gray.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User / CLI Layer
    subgraph "User / CLI Layer"
        CLI["mmpdb CLI"]:::cli
        fragment["fragment"]:::cli
        index["index"]:::cli
        loadprops["loadprops"]:::cli
        transform["transform"]:::cli
        predict["predict"]:::cli
        generate["generate"]:::cli
        help["help"]:::cli
        smi_split["smi_split"]:::cli
        fragdb_partition["fragdb_partition"]:::cli
        fragdb_merge["fragdb_merge"]:::cli
    end

    %% Core Library Modules
    subgraph "Core Library Modules"
        fileio["fileio"]:::core
        fragmentAlg["fragment_algorithm"]:::core
        analysisAlg["analysis_algorithms"]:::core
        rgroup2smarts["rgroup2smarts"]:::core
        smiles_syntax["smiles_syntax"]:::core
        cansmirks["cansmirks_table"]:::core
        dbutils["dbutils"]:::core
        schema["schema"]:::core
        index_algorithm["index_algorithm"]:::core
        index_types["index_types"]:::core
        index_writers["index_writers"]:::core
        properties_io["properties_io"]:::core
        reporters["reporters"]:::core
        environment["environment"]:::core
        fragment_db["fragment_db"]:::core
        fragment_records["fragment_records"]:::core
        fragment_types["fragment_types"]:::core
        click_utils["click_utils"]:::core
        smi_utils["smi_utils"]:::core
        config["config"]:::core
    end

    %% Storage & Cache
    subgraph "Storage & Cache"
        smiles_file["SMILES Files"]:::data
        fragdb[".fragdb (Flat File)"]:::data
        sqlite_db["SQLite / Postgres DB"]:::data
        csv_props["CSV Properties"]:::data
    end

    %% External Dependencies
    subgraph "External Dependencies"
        rdkit["RDKit"]:::external
        numpy_scipy["NumPy / SciPy"]:::external
        peewee["Peewee / psycopg2"]:::external
        click_lib["Click"]:::external
        psutil["psutil"]:::external
    end

    %% Data Flows
    smiles_file -->|SMILES| fragment
    fragment -->|calls| fileio
    fragment -->|calls| fragmentAlg
    fragmentAlg -->|produces| fragdb
    fragdb -->|input| index
    index -->|calls| index_algorithm
    index_algorithm -->|uses ORM| index_writers
    index_writers -->|writes| sqlite_db
    csv_props -->|loads| loadprops
    loadprops -->|calls| properties_io
    properties_io -->|updates| sqlite_db
    sqlite_db -->|queries| transform
    sqlite_db -->|queries| predict
    sqlite_db -->|queries| generate
    transform -->|uses| analysisAlg
    predict -->|uses| analysisAlg
    generate -->|uses| environment

    %% Distributed Utilities Flows
    smi_split -->|splits| smiles_file
    fragdb_partition -->|shards| fragdb
    fragdb_merge -->|merges| fragdb

    %% External integration
    fragmentAlg -->|RDKit ops| rdkit
    fileio -->|numpy/scipy| numpy_scipy
    index_algorithm -->|psutil| psutil
    index_algorithm -->|db access| peewee
    fragment -->|CLI lib| click_lib
    index -->|CLI lib| click_lib
    loadprops -->|CLI lib| click_lib
    transform -->|CLI lib| click_lib
    predict -->|CLI lib| click_lib
    generate -->|CLI lib| click_lib
    smi_split -->|CLI lib| click_lib
    fragdb_partition -->|CLI lib| click_lib
    fragdb_merge -->|CLI lib| click_lib

    %% Click Events
    click CLI "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/__main__.py"
    click fragment "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/fragment.py"
    click index "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/index.py"
    click loadprops "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/loadprops.py"
    click transform "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/transform.py"
    click predict "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/predict.py"
    click generate "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/generate.py"
    click help "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/help_.py"
    click smi_split "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/smi_split.py"
    click fragdb_partition "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/fragdb_partition.py"
    click fragdb_merge "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/fragdb_merge.py"
    click fragmentAlg "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/fragment_algorithm.py"
    click analysisAlg "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/analysis_algorithms.py"
    click rgroup2smarts "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/rgroup2smarts.py"
    click smiles_syntax "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/smiles_syntax.py"
    click cansmirks "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cansmirks_table.py"
    click fileio "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/fileio.py"
    click dbutils "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/dbutils.py"
    click schema "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/schema.py"
    click index_algorithm "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/index_algorithm.py"
    click index_types "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/index_types.py"
    click index_writers "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/index_writers.py"
    click properties_io "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/properties_io.py"
    click reporters "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/reporters.py"
    click environment "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/environment.py"
    click fragment_db "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/fragment_db.py"
    click fragment_records "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/fragment_records.py"
    click fragment_types "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/fragment_types.py"
    click click_utils "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/click_utils.py"
    click smi_utils "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/cli/smi_utils.py"
    click config "https://github.com/rdkit/mmpdb/blob/master/mmpdblib/config.py"

    %% Styles
    classDef cli fill:#cce5ff,stroke:#004085,color:#004085;
    classDef core fill:#d4edda,stroke:#155724,color:#155724;
    classDef data fill:#fff3cd,stroke:#856404,color:#856404,stroke-dasharray: 5 5;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
```
