# lamalab-org/chempile architecture

> Architecture diagram of the GitHub repository lamalab-org/chempile, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lamalab-org/chempile
- Repository: https://github.com/lamalab-org/chempile

## Overview

##### Project type and purpose
- ChemPile (ChemNLP) is a Python library and CLI-based toolkit for building large, diverse chemistry datasets and training/evaluating LLMs on them.
- It provides:
- Data-ingestion & processing pipelines (tabular, text, knowledge-graph).
- CLI entrypoints for sampling, token counting, rephrasing.
- Experiment orchestration for LLM training (via GPT-NeoX submodule) and evaluation (via lm-eval harness).
- Documentation site (MkDocs) and CI/CD (GitHub Actions).

A. Raw data layer
- Local filesystem under data/ (tabular/, natural/, kg/)
- External sources: Hugging Face datasets, PubMed/EuropePMC, ChEMBL, UniProt, etc.
B. Processing pipelines
- src/chemnlp/data: converters, samplers, splitters, meta-yaml generators, tokenizer adapters
- data_val submodule: schema validation and unit tests
- Custom scripts (data/check_*.py, text_sampling/*, arxives/token_counter_cli.py)
C. Core library
- chemnlp/utils: shared helpers
- Versioned dataset definitions in hf_datasets.py
- Transform modules per dataset under data/tabular/*/transform.py
D. CLI interfaces
- chemnlp/data/sampler_cli.py → “chemnlp sample”
- token_counter_cli.py → “chemnlp count-tokens”
- rephrase_chempile/rephrase.py → “chemnlp rephrase”
E. Experiment & training orchestration
- experiments/ scripts invoke:
- Data preparation (experiments/data/*)
- Ablation & pretraining (experiments/ablations/*.py)
- Submission scripts to cluster (sbatch_*.sh)
- GPT-NeoX submodule (for model architecture & training)
- WandB integration for logging & hyperparameter tracking
- embedding_dataset/ for feature extraction
- lm-evaluation-harness & lm-eval2 for model evaluation
F. CI/CD & QA
- .github/workflows/install.yaml: automated lint/test on PRs
- pre-commit hooks (.pre-commit-config.yaml)
- tests/: pytest suite for sampler, reprs, NER
G. Documentation & packaging
- docs/ (MkDocs config mkdocs.yml)
- API docs under docs/api/
- pyproject.toml & conda.yaml for environment and packaging

##### Architectural patterns & design principles
- Layered architecture:
- Data layer (raw files, external APIs)
- Business logic layer (chemnlp.data pipelines, transformations)
- CLI/Application layer (entrypoints)
- Experiment layer (scripts that orchestrate training)
- Modularity per dataset: each dataset has its own directory, metadata YAML, transform script.
- Dependency injection: dataset functions accept tokenizer and return tokenized splits.
- Git submodules for large external components (gpt-neox, lm-eval).
- Standard Python packaging & testing conventions.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RD["Raw Data (data/)"]:::data
        click RD "https://github.com/lamalab-org/chempile/tree/main/data/"
        TD["Tabular Data (data/tabular/)"]:::data
        click TD "https://github.com/lamalab-org/chempile/tree/main/data/tabular/"
        ND["Natural Data (data/natural/)"]:::data
        click ND "https://github.com/lamalab-org/chempile/tree/main/data/natural/"
        KG["KG Data (data/kg/)"]:::data
        click KG "https://github.com/lamalab-org/chempile/tree/main/data/kg/"
        ES["External Sources\n(HF Datasets, PubMed, ChEMBL, UniProt)"]:::data
    end

    %% Processing Pipelines
    subgraph "Processing Pipelines"
        direction TB
        DP["Data Pipelines\n(src/chemnlp/data/)"]:::blue
        click DP "https://github.com/lamalab-org/chempile/tree/main/src/chemnlp/data/"
        DV["Data Validation\n(src/chemnlp/data_val/, tests/data/)"]:::blue
        click DV "https://github.com/lamalab-org/chempile/tree/main/src/chemnlp/data_val/"
        click DV "https://github.com/lamalab-org/chempile/tree/main/tests/data/"
        CDS["Custom Scripts\n(data/check_*.py, data/text_sampling/, data/postprocess_split.py)"]:::blue
        click CDS "https://github.com/lamalab-org/chempile/blob/main/data/check_pandas.py"
        click CDS "https://github.com/lamalab-org/chempile/blob/main/data/check_smiles_split.py"
        click CDS "https://github.com/lamalab-org/chempile/blob/main/data/postprocess_split.py"
        click CDS "https://github.com/lamalab-org/chempile/tree/main/data/text_sampling/"
        TCC["Token Counter CLI\n(arxives/token_counter_cli.py)"]:::blue
        click TCC "https://github.com/lamalab-org/chempile/blob/main/arxives/token_counter_cli.py"
    end

    %% Core Library & CLI
    subgraph "Core Library & CLI"
        direction TB
        UTIL["Utilities\n(src/chemnlp/utils.py)"]:::blue
        click UTIL "https://github.com/lamalab-org/chempile/blob/main/src/chemnlp/utils.py"
        HF["HF Datasets Defs\n(src/chemnlp/data/hf_datasets.py)"]:::blue
        click HF "https://github.com/lamalab-org/chempile/blob/main/src/chemnlp/data/hf_datasets.py"
        TF["Tabular Transforms\n(data/tabular/*/transform.py)"]:::blue
        click TF "https://github.com/lamalab-org/chempile/tree/main/data/tabular/"
        CLI1["Sampler CLI\n(src/chemnlp/data/sampler_cli.py)"]:::blue
        click CLI1 "https://github.com/lamalab-org/chempile/blob/main/src/chemnlp/data/sampler_cli.py"
        CLI2["Rephrase CLI\n(rephrase_chempile/rephrase.py)"]:::blue
        click CLI2 "https://github.com/lamalab-org/chempile/blob/main/rephrase_chempile/rephrase.py"
    end

    %% Experiment & Training
    subgraph "Experiment & Training"
        direction TB
        PREP["Data Prep\n(experiments/data/)"]:::orange
        click PREP "https://github.com/lamalab-org/chempile/tree/main/experiments/data/"
        ABLA["Ablations & Pretraining\n(experiments/ablations/)"]:::orange
        click ABLA "https://github.com/lamalab-org/chempile/tree/main/experiments/ablations/"
        SBAT["Cluster Scripts\n(experiments/data/sbatch_*.sh)"]:::orange
        click SBAT "https://github.com/lamalab-org/chempile/blob/main/experiments/data/sbatch_*.sh"
        GNX["GPT-NeoX Submodule"]:::orange
        EB["Embedding Dataset\n(embedding_dataset/)"]:::orange
        click EB "https://github.com/lamalab-org/chempile/tree/main/embedding_dataset/"
        MEVAL["Model Eval\n(lm-eval2/, lm-evaluation-harness/)"]:::orange
        click MEVAL "https://github.com/lamalab-org/chempile/tree/main/lm-eval2/"
        click MEVAL "https://github.com/lamalab-org/chempile/tree/main/lm-evaluation-harness/"
        WB["WandB"]:::orange
    end

    %% Supporting Infrastructure
    subgraph "Supporting Infrastructure"
        direction TB
        CI["CI/CD\n(.github/workflows/install.yaml)"]:::gray
        click CI "https://github.com/lamalab-org/chempile/blob/main/.github/workflows/install.yaml"
        PC["Pre-commit\n(.pre-commit-config.yaml)"]:::gray
        click PC "https://github.com/lamalab-org/chempile/blob/main/.pre-commit-config.yaml"
        TEST["Test Suite\n(tests/)"]:::gray
        click TEST "https://github.com/lamalab-org/chempile/tree/main/tests/"
        DOCS["Docs (MkDocs)\n(docs/, mkdocs.yml)"]:::gray
        click DOCS "https://github.com/lamalab-org/chempile/tree/main/docs/"
        click DOCS "https://github.com/lamalab-org/chempile/blob/main/mkdocs.yml"
        APIDOC["API Docs\n(docs/api/)"]:::gray
        click APIDOC "https://github.com/lamalab-org/chempile/tree/main/docs/api/"
        PKG["Packaging\n(pyproject.toml, conda.yaml)"]:::gray
        click PKG "https://github.com/lamalab-org/chempile/blob/main/pyproject.toml"
        click PKG "https://github.com/lamalab-org/chempile/blob/main/conda.yaml"
    end

    %% Connections
    RD --> DP
    TD --> DP
    ND --> DP
    KG --> DP
    ES --> DP

    DP --> UTIL

    CLI1 --> DP
    TCC --> UTIL
    CLI2 --> UTIL

    UTIL --> PREP
    HF --> PREP
    TF --> PREP

    PREP --> ABLA
    ABLA --> GNX
    GNX --> WB
    GNX --> EB
    EB --> MEVAL
    MEVAL --> WB

    CI --> TEST
    CI --> DOCS

    %% Styles
    classDef data fill:#ccffcc,stroke:#333,stroke-width:1px;
    classDef blue fill:#cce5ff,stroke:#333,stroke-width:1px;
    classDef orange fill:#ffe0cc,stroke:#333,stroke-width:1px;
    classDef gray fill:#f2f2f2,stroke:#333,stroke-width:1px;
```
