How lamalab-org/chempile works
Project type and purpose
• ChemPile (ChemNLP) is a Python library and CLI-based toolkit for building large, diverse chemistry datasets and training/evaluating LLMs on them.
• It provides:
• Data-ingestion & processing pipelines (tabular, text, knowledge-graph).
• CLI entrypoints for sampling, token counting, rephrasing.
• Experiment orchestration for LLM training (via GPT-NeoX submodule) and evaluation (via lm-eval harness).
• Documentation site (MkDocs) and CI/CD (GitHub Actions).
A. Raw data layer
• Local filesystem under data/ (tabular/, natural/, kg/)
• External sources: Hugging Face datasets, PubMed/EuropePMC, ChEMBL, UniProt, etc.
B. Processing pipelines
• src/chemnlp/data: converters, samplers, splitters, meta-yaml generators, tokenizer adapters
• data_val submodule: schema validation and unit tests
• Custom scripts (data/check_*.py, text_sampling/*, arxives/token_counter_cli.py)
C. Core library
• chemnlp/utils: shared helpers
• Versioned dataset definitions in hf_datasets.py
• Transform modules per dataset under data/tabular/*/transform.py
D. CLI interfaces
• chemnlp/data/sampler_cli.py → “chemnlp sample”
• token_counter_cli.py → “chemnlp count-tokens”
• rephrase_chempile/rephrase.py → “chemnlp rephrase”
E. Experiment & training orchestration
• experiments/ scripts invoke:
• Data preparation (experiments/data/*)
• Ablation & pretraining (experiments/ablations/*.py)
• Submission scripts to cluster (sbatch_*.sh)
• GPT-NeoX submodule (for model architecture & training)
• WandB integration for logging & hyperparameter tracking
• embedding_dataset/ for feature extraction
• lm-evaluation-harness & lm-eval2 for model evaluation
F. CI/CD & QA
• .github/workflows/install.yaml: automated lint/test on PRs
• pre-commit hooks (.pre-commit-config.yaml)
• tests/: pytest suite for sampler, reprs, NER
G. Documentation & packaging
• docs/ (MkDocs config mkdocs.yml)
• API docs under docs/api/
• pyproject.toml & conda.yaml for environment and packaging
Architectural patterns & design principles
• Layered architecture:
• Data layer (raw files, external APIs)
• Business logic layer (chemnlp.data pipelines, transformations)
• CLI/Application layer (entrypoints)
• Experiment layer (scripts that orchestrate training)
• Modularity per dataset: each dataset has its own directory, metadata YAML, transform script.
• Dependency injection: dataset functions accept tokenizer and return tokenized splits.
• Git submodules for large external components (gpt-neox, lm-eval).
• Standard Python packaging & testing conventions.