# marin-community/marin architecture

> Architecture diagram of the GitHub repository marin-community/marin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marin-community/marin
- Repository: https://github.com/marin-community/marin

## Overview

##### Project Type and Purpose
- Marin is a full-stack, Python-based research framework for building, running, and reproducing large-scale language-model experiments.
- It combines a core library (`marin/`), a suite of experiment definitions (`experiments/`), a standalone data‐browser web app (`data_browser/`), infrastructure templates (`infra/`), tutorials/docs (`docs/`), and CI/deployment scripts (`docker/`, GitHub workflows, Makefile).

##### Top-Level Components
A. Core Library (`marin/`):
- Execution engine (DAG scheduler) in `marin/execution/`
- Data‐ingest modules: `marin/download/`, `marin/datashop/`, `marin/processing/`, `marin/transform/`
- Model training & inference: `marin/training/`, `marin/generation/`
- Evaluation suite: `marin/evaluation/`
- Utilities, schemas, tokenization support
B. Experiments (`experiments/`):
- Python scripts defining “steps” that wire together core‐library modules.
- Default configs, special pipelines (e.g. datashop, DCLM, speedrun).
C. Data Browser (`data_browser/`):
- React frontend (`src/`) served by a Flask backend (`server.py`, `wsgi.py`).
- Configuration for local/GCP/Dockerized deployment (`docker-compose.yml`, `Dockerfile.prod`).
- Provides a UI to explore experiment outputs and datasets.
D. Infrastructure (`infra/` + `docker/`):
- Cluster run configs (`infra/*.yaml`), TPU monitoring, Ray launcher
- Docker build contexts for “levanter” (LM eval), “marin” clusters.
- CI/CD workflows in `.github/workflows/`, container builds, lint, tests.
E. Documentation & Tutorials (`docs/` + `README.md`):
- MkDocs config, guides, API references, example retrospectives.
F. Tests & CI (`tests/` + GitHub Actions):
- Unit, integration, snapshot tests for core modules and data pipelines.
G. Utility Scripts (`scripts/`):
- Helper scripts for reorg, summary generation, Ray cluster ops.

##### Architectural Patterns
- Directed‐acyclic‐graph orchestration: each experiment step declares dependencies, executed by `marin.execution.executor`.
- Modular pipeline stages: download→processing→transform→tokenize→train→evaluate→export.
- Micro‐services style: Data Browser runs as a separate service, communicating over HTTP to fetch metadata.
- Containerized deployments: Dockerfiles for each service, Helm/YAML for clusters.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Entry Points
    subgraph "Entry Points"
        direction TB
        CLI["Makefile / CLI / run-dev.py"]:::code
        RAY["ray_run.py / slurm_run.py"]:::code
    end

    %% Marin Library
    subgraph "Marin Library" 
        direction TB
        Exec["Execution Engine"]:::code
        Download["Data Ingestion: Download"]:::code
        Datashop["Data Ingestion: Datashop"]:::code
        Processing["Processing"]:::code
        Transform["Transform"]:::code
        Tokenize["Tokenization Support"]:::code
        Train["Model Training"]:::code
        Infer["Inference / Generation"]:::code
        Eval["Evaluation Suite"]:::code
        Export["Export / HF Upload"]:::code
    end

    %% Experiments
    Experiments["Experiments Layer"]:::code

    %% Data Browser
    subgraph "Data Browser"
        direction TB
        Frontend["React Frontend"]:::code
        Backend["Flask Backend"]:::code
    end

    %% Storage
    subgraph "Storage Endpoints"
        direction TB
        GCS["GCS / Local FS"]:::storage
        HFHub["Hugging Face Hub"]:::storage
        WandB["Weights & Biases"]:::storage
    end

    %% Compute & Infra
    Compute["Compute Cluster (Ray/Slurm/TPU)"]:::infra
    CICD["CI/CD Pipeline"]:::infra
    Infra["Infra Templates & Docker"]:::infra

    %% Flow & Relationships
    CLI -->|“invoke”| Exec
    RAY -->|“invoke”| Exec
    Experiments -->|“defines steps”| Exec
    Exec -->|“download data”| Download
    Exec -->|“ingest via datashop”| Datashop
    Download --> Processing
    Datashop --> Processing
    Processing --> Transform
    Transform --> Tokenize
    Tokenize --> Train
    Train --> Infer
    Infer --> Eval
    Eval --> Export
    Exec -->|“dispatch tasks”| Compute
    Train -->|“log metrics”| WandB
    Eval -->|“log metrics”| WandB
    Export -->|“push models”| HFHub
    Export -->|“store artifacts”| GCS
    Backend -->|HTTP| GCS
    Backend -->|HTTP| HFHub
    Frontend -->|HTTP| Backend
    CICD ---|“triggers tests and builds”| CLI
    CICD --- Infra
    CICD ---|“deploy Data Browser”| Backend

    %% Click Events
    click Exec "https://github.com/marin-community/marin/blob/main/marin/execution/executor.py"
    click Download "https://github.com/marin-community/marin/blob/main/marin/download/huggingface/download.py"
    click Datashop "https://github.com/marin-community/marin/blob/main/marin/datashop/pipeline.py"
    click Processing "https://github.com/marin-community/marin/blob/main/marin/processing/tokenize/tokenize.py"
    click Transform "https://github.com/marin-community/marin/blob/main/marin/transform/ar5iv/transform_ar5iv.py"
    click Tokenize "https://github.com/marin-community/marin/blob/main/marin/processing/tokenize/tokenize.py"
    click Train "https://github.com/marin-community/marin/blob/main/marin/training/training.py"
    click Infer "https://github.com/marin-community/marin/blob/main/marin/generation/inference.py"
    click Eval "https://github.com/marin-community/marin/blob/main/marin/evaluation/run.py"
    click Export "https://github.com/marin-community/marin/blob/main/marin/export/hf_upload.py"
    click Experiments "https://github.com/marin-community/marin/blob/main/experiments/exp780_default_suite.py"
    click Frontend "https://github.com/marin-community/marin/blob/main/data_browser/src/App.js"
    click Backend "https://github.com/marin-community/marin/blob/main/data_browser/server.py"
    click Infra "https://github.com/marin-community/marin/blob/main/infra/marin-cluster-template.yaml"
    click CICD "https://github.com/marin-community/marin/blob/main/.github/workflows/unit-tests.yaml"

    %% Styles
    classDef code fill:#D0E6FF,stroke:#0288D1,color:#000;
    classDef storage fill:#E8F5E9,stroke:#2E7D32,color:#000;
    classDef infra fill:#FFF3E0,stroke:#EF6C00,color:#000;
```
