How rdkit/mmpdb works
Project Type and Purpose
• This is a command‐line, open-source cheminformatics toolkit (“mmpdb”) written in Python.
• It implements a pipeline for (a) fragmenting molecular structures (SMILES), (b) indexing matched molecular pairs into a SQLite (or Postgres) database, and (c) analyzing or generating new molecules based on rule transformations and property predictions.
High-Level Components
a. User / CLI Layer
• The mmpdb executable, implemented with the Click library.
• Subcommands: fragment, index, loadprops, transform, predict, generate, plus help-* and distributed utilities (smi_split, fragdb_partition, fragdb_merge).
b. Core Library (mmpdblib/)
• CLI dispatch (cli/… modules)
• Fragmentation engines (fragment_algorithm.py, analysis_algorithms.py, rgroup2smarts.py, smiles_syntax.py)
• File I/O (fileio.py) – reading SMILES, writing fragment databases (.fragdb files).
• Database utilities (dbutils.py, schema.py, SQL scripts in .sql files).
• Indexing engines (index_algorithm.py, index_types.py, index_writers.py) using the Peewee ORM.
• Property I/O (properties_io.py) and reporting (reporters.py).
• Environment/fingerprint generation (environment.py, cansmirks_table.py).
c. Data Stores & Formats
• Flat‐file fragment database: .fragdb (binary or JSON‐like).
• SQLite (or Postgres) MMP database .mmpdb with tables: compounds, rule_smiles, rules, environments, rule_stats, pairs.
• Optional cache files for fragmentation.
d. External Dependencies
• RDKit for cheminformatics operations
• NumPy, SciPy for data and statistics
• Click for CLI
• Peewee (and optional psycopg2) for database access
• psutil for memory metrics in index
Interactions and Data Flow
1) User invokes mmpdb fragment → CLI module picks up SMILES file → calls fileio + fragment_algorithm → produces a .fragdb file (with optional caching).
2) User invokes mmpdb index → CLI module loads .fragdb → index_algorithm groups by constant fragments → writes SQLite .mmpdb via Peewee.
3) mmpdb loadprops → reads CSV properties → updates .mmpdb tables and recomputes rule statistics.
4) Query commands (transform, predict) → read .mmpdb → analysis_algorithms compute matched pair transformations and property shifts → output to console or file.
5) mmpdb generate → leverages rules + environments → produces new SMILES suggestions.
6) Distributed utilities (smi_split, fragdb_partition, fragdb_merge) operate on .smi or .fragdb shards in parallel, then merge results.
Architectural Patterns and Principles
• Modular CLI with subcommand pattern (Click).
• Separation of concerns: I/O, core algorithms, database access are isolated in separate modules.
• Data-driven pipeline: input files → intermediate flat files → relational DB → analysis.
• Use of caching and partitioning to scale large datasets.
• Lightweight ORM for portability (SQLite + optional Postgres).
• User / CLI Layer
• Core Library Modules
• Storage & Cache (flat-files, DB)
• External Services / Libraries
• “mmpdb CLI (Click)” at top.
• From SMILES input → Fragmenter → .fragdb
• From .fragdb → Indexer → SQLite DB
• From CSV Properties → Property Loader → SQLite DB
• From SQLite DB → Analyzer / Transformer / Generator → console or output file
• From Fragment & Index → optional partition/merge utilities → merged fragments → index
• Core algorithm modules in another (green).
• External libraries in gray.