# pku-alignment/align-anything architecture

> Architecture diagram of the GitHub repository pku-alignment/align-anything, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pku-alignment/align-anything
- Repository: https://github.com/pku-alignment/align-anything

## Overview

The align-anything project is a highly modular, multi-modal alignment framework designed to fine-tune and evaluate “any-to-any” models that span various modalities (text, image, audio, video, and even actions). Its primary purpose is to provide a unified pipeline that supports training, evaluation, and interactive serving (via CLI, multi-modal CLI, and an interactive arena), all based on flexible algorithms (such as SFT, DPO, PPO, RM, etc.) with extensive support for metering modality-specific datasets and benchmarks.

##### Main Components
- Core Framework Package:
– The “align_anything” package is the heart of the system. It includes several sub-modules:
- Architecture: Contains core agents and model definitions (see align_anything/architecture, which includes transformer models, image encoders, and preprocessors).
- Configs: Houses numerous configuration files and templates (for training, evaluation, deep speed, vLLM, benchmark settings, etc.).
- Datasets: Provides code to process various modalities – from text-only to multi-modal formats – and to format samples for supervised, preference, or prompt-only training.
- Environment: Includes utilities and sensor modules for simulated or real-world data (navigation, vision, manipulation, etc.).
- Evaluation: Contains scripts, benchmarks, and evaluation pipelines (with separated benchmark directories and separate handling of inference approaches such as vLLM and DeepSpeed).
- Models: Includes model implementations and registries that support a wide range of pre-built models as well as custom options.
- Trainers: Hosts training pipelines split by modality and algorithm (SFT, DPO, PPO, etc.) for different training tasks. Notice further directories per modality (text-to-text, text-to-image, among others).
- Serve: Provides components for interactive inference (CLI, multi-modal CLI, arena) and model workers.
- Utils: Contains helper functions, logging, device handling, and more.

- Scripts and Deployment:
– The “scripts” directory contains automation scripts for training (including modality-specific shell scripts), Docker container setup, evaluation, model comparisons, and more.
– The “Dockerfile” and related CI/Contributing documents under “.github” indicate CI integration and potential automation for deployment.

- Projects and Cookbooks:
– Example projects, demonstrations, and tutorials for extending and using the framework (e.g., projects/janus, projects/lang_feedback) are available.

##### Relationships and Data Flows
- Data Intake & Preprocessing:

- Training Pipeline:
– Trainers take in formatted examples, utilize model definitions (from Models and Architecture) and configurations (from Configs) to perform fine-tuning using various algorithms. Label these pipelines (SFT, DPO, PPO, etc.) clearly and show directional flows from configs/datasets to trainers to models.

- Evaluation and Benchmarking:

- Inference & Serving:

- Configuration and External Integration:
– The “configs” directory centralizes experiment parameters for DeepSpeed, vLLM, training, evaluation, etc. Indicate that these config files are consumed by Trainers, Evaluation, and even Serve during runtime.
– Also highlight integration with external tools (Docker, Slurm for cluster training, and Wandb for logging).

##### Architectural Patterns and Design Principles
- Modularity:

- Flexible Pipelines:

- Reusability:
– Indicate that many components (configuration, dataset formatters, and evaluation scripts) are designed to be customizable or extendable for user-specified models and datasets.

##### Final Advice
- Keep it detailed: Reflect each major folder or logical group of functions.
- Clarity: Even though the framework is vast, aim to present a high-level overview that can be “drilled down” further if needed.
- Relationships are clear: For example, show how configuration files affect training and evaluation, and how datasets flow into the training pipelines.
- Do not overcomplicate: While it is important to capture essential architectural elements, keep components grouped by domain (e.g., group all training-related entities together).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Core Engine - align_anything"
        Architect["Architecture: Core Agents & Models"]:::training
        Configs["Configs: Training & Evaluation Templates"]:::training
        Datasets["Datasets & Formatters"]:::training
        Environment["Environment: Sensors & Data Utilities"]:::training
        Evaluation["Evaluation & Benchmarks"]:::evaluation
        Models["Models & Registries"]:::training
        Trainers["Trainers (SFT,DPO,PPO,etc.)"]:::training
        Serve["Serve: Interactive CLI & Inference"]:::serving
        Utils["Utils & Helpers"]:::training
    end

    subgraph "External Integration"
        Scripts["Automation Scripts"]:::external
        DockerfileNode["Dockerfile"]:::external
        GitHub[".github (CI/Contributing)"]:::external
    end

    subgraph "Projects & Cookbooks"
        Projects["Example Projects & Demos"]:::external
        Cookbooks["Tutorials & Cookbooks"]:::external
    end

    %% Core Engine Relationships
    Datasets -->|"data_formatted"| Trainers
    Configs -->|"training_config"| Trainers
    Configs -->|"evaluation_config"| Evaluation
    Trainers -->|"trained_models"| Models
    Evaluation -->|"benchmark_inference"| Models
    Serve -->|"inference_calls"| Models
    Architect -->|"defines"| Models

    Utils --- Architect
    Utils --- Configs
    Utils --- Datasets
    Utils --- Environment
    Utils --- Evaluation
    Utils --- Models
    Utils --- Trainers
    Utils --- Serve

    %% External Integration Relationships
    Scripts -->|"triggers"| Trainers
    Scripts -->|"triggers"| Evaluation
    Scripts -->|"manages"| Serve
    DockerfileNode -->|"automated_deployment"| Trainers
    GitHub -->|"automated_deployment"| Evaluation

    %% Projects & Cookbooks Relationships
    Projects -->|"demo_usage"| Architect
    Cookbooks -->|"tutorials"| Configs

    classDef training fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef evaluation fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef serving fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef external fill:#f8d7da,stroke:#721c24,stroke-width:2px;

    click Architect "https://github.com/pku-alignment/align-anything/tree/main/align_anything/architecture"
    click Configs "https://github.com/pku-alignment/align-anything/tree/main/align_anything/configs"
    click Datasets "https://github.com/pku-alignment/align-anything/tree/main/align_anything/datasets"
    click Environment "https://github.com/pku-alignment/align-anything/tree/main/align_anything/environment"
    click Evaluation "https://github.com/pku-alignment/align-anything/tree/main/align_anything/evaluation"
    click Models "https://github.com/pku-alignment/align-anything/tree/main/align_anything/models"
    click Trainers "https://github.com/pku-alignment/align-anything/tree/main/align_anything/trainers"
    click Serve "https://github.com/pku-alignment/align-anything/tree/main/align_anything/serve"
    click Utils "https://github.com/pku-alignment/align-anything/tree/main/align_anything/utils"
    click Scripts "https://github.com/pku-alignment/align-anything/tree/main/scripts"
    click DockerfileNode "https://github.com/pku-alignment/align-anything/tree/main/Dockerfile"
    click GitHub "https://github.com/pku-alignment/align-anything/blob/main/.github"
    click Projects "https://github.com/pku-alignment/align-anything/tree/main/projects"
    click Cookbooks "https://github.com/pku-alignment/align-anything/tree/main/cookbooks"
```
