How marin-community/marin works
Project Type and Purpose
• Marin is a full-stack, Python-based research framework for building, running, and reproducing large-scale language-model experiments.
• It combines a core library (marin/), a suite of experiment definitions (experiments/), a standalone data‐browser web app (data_browser/), infrastructure templates (infra/), tutorials/docs (docs/), and CI/deployment scripts (docker/, GitHub workflows, Makefile).
Top-Level Components
A. Core Library (marin/):
• Execution engine (DAG scheduler) in marin/execution/
• Data‐ingest modules: marin/download/, marin/datashop/, marin/processing/, marin/transform/
• Model training & inference: marin/training/, marin/generation/
• Evaluation suite: marin/evaluation/
• Utilities, schemas, tokenization support
B. Experiments (experiments/):
• Python scripts defining “steps” that wire together core‐library modules.
• Default configs, special pipelines (e.g. datashop, DCLM, speedrun).
C. Data Browser (data_browser/):
• React frontend (src/) served by a Flask backend (server.py, wsgi.py).
• Configuration for local/GCP/Dockerized deployment (docker-compose.yml, Dockerfile.prod).
• Provides a UI to explore experiment outputs and datasets.
D. Infrastructure (infra/ + docker/):
• Cluster run configs (infra/*.yaml), TPU monitoring, Ray launcher
• Docker build contexts for “levanter” (LM eval), “marin” clusters.
• CI/CD workflows in .github/workflows/, container builds, lint, tests.
E. Documentation & Tutorials (docs/ + README.md):
• MkDocs config, guides, API references, example retrospectives.
F. Tests & CI (tests/ + GitHub Actions):
• Unit, integration, snapshot tests for core modules and data pipelines.
G. Utility Scripts (scripts/):
• Helper scripts for reorg, summary generation, Ray cluster ops.
Architectural Patterns
• Directed‐acyclic‐graph orchestration: each experiment step declares dependencies, executed by marin.execution.executor.
• Modular pipeline stages: download→processing→transform→tokenize→train→evaluate→export.
• Micro‐services style: Data Browser runs as a separate service, communicating over HTTP to fetch metadata.
• Containerized deployments: Dockerfiles for each service, Helm/YAML for clusters.