# open-dataflow/rare architecture

> Architecture diagram of the GitHub repository open-dataflow/rare, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/open-dataflow/rare
- Repository: https://github.com/open-dataflow/rare

## Overview

##### Main components
a. Data Layer
- Raw datasets (data/…)
- Processed data (outputs of process/*.py)
b. Preprocessing Module
- process/*.py scripts (process_medqa.py, process_pubmed.py, etc.)
- select_true.py (filtering distilled outputs)
c. Distillation / Retrieval-Augmented Inference
- inference/vllm_infer_text.py (text-only distillation)
- inference/vllm_infer_mm.py (multi-modal distillation)
- inference/api_infer_post.py (external API calls)
d. Training Module
- train/train.py & train/sft.py (model fine-tuning)
- config files (accelerate_config.yaml, fsdp_config_*.json, training_args*.yaml)
- demo scripts (demo/*.sh) for end-to-end runs
e. Model Storage & Conversion
- saves/… (checkpoint directories)
- zero_to_fp32.py (converting Deepspeed to FP32)
f. Inference & Serving
- inference scripts reused for model serving
- optional API layer (wrap vllm_infer_text.py)
g. Evaluation Module
- eval/eval.py (compute accuracy, extract predictions)
h. External Services & Dependencies
- Hugging Face Datasets & Models (downloads)
- ModelScope downloads
- Accelerate, Deepspeed, VLLM libraries

##### Relationships and data flow
- Raw Data → Preprocessing Module → Processed JSON files
- Processed JSON → Distillation Module (via LLM) → “predict” fields
- Distilled JSON → select_true.py → Final Training Dataset
- Training Dataset + Model Configs → Training Module → Saved Checkpoints
- Saved Checkpoints → Inference Module / API → Predictions
- Predictions + Ground Truth → Evaluation Module → Metrics

##### Architectural patterns
- Pipeline Pattern: clearly separate data ingestion, distillation, training, inference, and evaluation.
- Modular Scripts: each folder (process/, train/, inference/, eval/) is an independent stage.
- Config-Driven: external YAML/JSON configs drive behavior (e.g., accelerate, FSDP).
- Batch vs. Online: preprocessing & training are batch workflows; API inference can be treated as a microservice.

##### Technologies
- Python (core scripts)
- Accelerate & Deepspeed (distributed training)
- VLLM (high-throughput LLM inference)
- Hugging Face & ModelScope (model & dataset hosting)
- Bash scripts (demo/ & train/sft.sh)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        DL_Data["dataset_info.json"]:::data
    end

    %% Preprocessing
    subgraph "Preprocessing Module"
        PM1["process_medqa.py"]:::module
        PM2["process_pubmed.py"]:::module
        PM3["process_mmrait.py"]:::module
        PM4["process_pubhealth.py"]:::module
        PM5["process_casehold.py"]:::module
        PM6["select_true.py"]:::module
        Proc_Out["Processed JSON"]:::data
    end

    %% Distillation / Retrieval-Augmented Inference
    subgraph "Distillation / Inference Module"
        DI1["vllm_infer_text.py"]:::module
        DI2["vllm_infer_mm.py"]:::module
        DI3["api_infer_post.py"]:::module
        Dist_JSON["Distilled JSON"]:::data
    end

    %% Training Module
    subgraph "Training Module"
        TM1["train.py"]:::module
        TM2["sft.py"]:::module
        CFG1["accelerate_config.yaml"]:::config
        CFG2["fsdp_config_llama.json"]:::config
        CFG3["fsdp_config_mistral.json"]:::config
        CFG4["fsdp_config_qwen.json"]:::config
        CFG5["training_args.yaml"]:::config
        CFG6["training_args_mm.yaml"]:::config
        Checkpoints["Model Checkpoints"]:::data
    end

    %% Demo Scripts
    subgraph "Demo Scripts"
        Demo1["llama_pubmedqa_rare.sh"]:::script
        Demo2["qwenvl_mmrait_rare.sh"]:::script
    end

    %% Inference & Serving
    subgraph "Inference & Serving"
        IS1["vllm_infer_text.py"]:::module
        IS2["vllm_infer_mm.py"]:::module
        IS3["api_infer_post.py"]:::module
        Predictions["Predictions (JSON)"]:::data
    end

    %% Evaluation Module
    subgraph "Evaluation Module"
        Eval["eval.py"]:::module
        Metrics["Metrics"]:::data
    end

    %% External Services
    subgraph "External Services"
        HF["Hugging Face Datasets & Models"]:::ext
        MS["ModelScope"]:::ext
        ACC["Accelerate/Deepspeed/VLLM"]:::ext
    end

    %% Connections
    DL_Data -->|"raw data JSON"| PM1
    DL_Data --> PM2
    DL_Data --> PM3
    DL_Data --> PM4
    DL_Data --> PM5
    PM1 -->|"processed JSON"| Proc_Out
    PM2 --> Proc_Out
    PM3 --> Proc_Out
    PM4 --> Proc_Out
    PM5 --> Proc_Out

    Proc_Out --> DI1
    Proc_Out --> DI2
    Proc_Out --> DI3
    DI1 -->|"distilled JSON"| Dist_JSON
    DI2 --> Dist_JSON
    DI3 --> Dist_JSON

    Dist_JSON --> PM6
    PM6 -->|"final training dataset"| Checkpoints

    Checkpoints --> IS1
    Checkpoints --> IS2
    Checkpoints --> IS3
    IS1 --> Predictions
    IS2 --> Predictions
    IS3 --> Predictions

    Predictions --> Eval
    DL_Data -.-> HF
    DI1 -.-> HF
    DI2 -.-> HF
    DI3 -.-> MS
    TM1 -.-> ACC
    TM2 -.-> ACC

    Demo1 --> Checkpoints
    Demo2 --> Checkpoints

    %% Click Events
    click DL_Data "https://github.com/open-dataflow/rare/blob/main/data/dataset_info.json"
    click PM1 "https://github.com/open-dataflow/rare/blob/main/process/process_medqa.py"
    click PM2 "https://github.com/open-dataflow/rare/blob/main/process/process_pubmed.py"
    click PM3 "https://github.com/open-dataflow/rare/blob/main/process/process_mmrait.py"
    click PM4 "https://github.com/open-dataflow/rare/blob/main/process/process_pubhealth.py"
    click PM5 "https://github.com/open-dataflow/rare/blob/main/process/process_casehold.py"
    click PM6 "https://github.com/open-dataflow/rare/blob/main/process/select_true.py"
    click DI1 "https://github.com/open-dataflow/rare/blob/main/inference/vllm_infer_text.py"
    click DI2 "https://github.com/open-dataflow/rare/blob/main/inference/vllm_infer_mm.py"
    click DI3 "https://github.com/open-dataflow/rare/blob/main/inference/api_infer_post.py"
    click TM1 "https://github.com/open-dataflow/rare/blob/main/train/train.py"
    click TM2 "https://github.com/open-dataflow/rare/blob/main/train/sft.py"
    click CFG1 "https://github.com/open-dataflow/rare/blob/main/train/accelerate_config.yaml"
    click CFG2 "https://github.com/open-dataflow/rare/blob/main/train/fsdp_config_llama.json"
    click CFG3 "https://github.com/open-dataflow/rare/blob/main/train/fsdp_config_mistral.json"
    click CFG4 "https://github.com/open-dataflow/rare/blob/main/train/fsdp_config_qwen.json"
    click CFG5 "https://github.com/open-dataflow/rare/blob/main/train/training_args.yaml"
    click CFG6 "https://github.com/open-dataflow/rare/blob/main/train/training_args_mm.yaml"
    click Demo1 "https://github.com/open-dataflow/rare/blob/main/demo/llama_pubmedqa_rare.sh"
    click Demo2 "https://github.com/open-dataflow/rare/blob/main/demo/qwenvl_mmrait_rare.sh"
    click Eval "https://github.com/open-dataflow/rare/blob/main/eval/eval.py"

    %% Styles
    classDef module fill:#bae1ff,stroke:#1f78b4;
    classDef data fill:#baffc9,stroke:#33a02c;
    classDef ext fill:#fdbf6f,stroke:#ff7f00;
    classDef config fill:#ffffb3,stroke:#b15928;
    classDef script fill:#fb9a99,stroke:#e31a1c;
```
