How open-dataflow/rare works
Main components
a. Data Layer
• Raw datasets (data/…)
• Processed data (outputs of process/*.py)
b. Preprocessing Module
• process/*.py scripts (process_medqa.py, process_pubmed.py, etc.)
• select_true.py (filtering distilled outputs)
c. Distillation / Retrieval-Augmented Inference
• inference/vllm_infer_text.py (text-only distillation)
• inference/vllm_infer_mm.py (multi-modal distillation)
• inference/api_infer_post.py (external API calls)
d. Training Module
• train/train.py & train/sft.py (model fine-tuning)
• config files (accelerate_config.yaml, fsdp_config_*.json, training_args*.yaml)
• demo scripts (demo/*.sh) for end-to-end runs
e. Model Storage & Conversion
• saves/… (checkpoint directories)
• zero_to_fp32.py (converting Deepspeed to FP32)
f. Inference & Serving
• inference scripts reused for model serving
• optional API layer (wrap vllm_infer_text.py)
g. Evaluation Module
• eval/eval.py (compute accuracy, extract predictions)
h. External Services & Dependencies
• Hugging Face Datasets & Models (downloads)
• ModelScope downloads
• Accelerate, Deepspeed, VLLM libraries
Relationships and data flow
• Raw Data → Preprocessing Module → Processed JSON files
• Processed JSON → Distillation Module (via LLM) → “predict” fields
• Distilled JSON → select_true.py → Final Training Dataset
• Training Dataset + Model Configs → Training Module → Saved Checkpoints
• Saved Checkpoints → Inference Module / API → Predictions
• Predictions + Ground Truth → Evaluation Module → Metrics
Architectural patterns
• Pipeline Pattern: clearly separate data ingestion, distillation, training, inference, and evaluation.
• Modular Scripts: each folder (process/, train/, inference/, eval/) is an independent stage.
• Config-Driven: external YAML/JSON configs drive behavior (e.g., accelerate, FSDP).
• Batch vs. Online: preprocessing & training are batch workflows; API inference can be treated as a microservice.
Technologies
• Python (core scripts)
• Accelerate & Deepspeed (distributed training)
• VLLM (high-throughput LLM inference)
• Hugging Face & ModelScope (model & dataset hosting)
• Bash scripts (demo/ & train/sft.sh)