How willxxy/ecg-byte works
Data Sources and Ingestion
• Raw ECG Data: Represent external datasets (PTB-XL, MIMIC-IV and ECG-QA). Note that the README instructs users to download and prepare data in a “data” folder.
• External Mapping: There is a mapping process that aligns the PTB-XL/MIMIC data with ECG-QA (using scripts provided in the README and instructions from other repositories).
• Pretrain Data: Also include data curated by ECG-Chat for the pretrain stage.
Preprocessing Pipeline
• Preprocess folder: Contains scripts (e.g. preprocess.sh) and Python modules in “preprocess” that load raw ECG files, handle segments (e.g., segment lengths such as 2500 samples) and apply transformations (including PCA).
• Sampling: The “sample_ecg.sh” script is used to sample the ECGs and produce a text file used for training the tokenizer.
• Data Loader: “data_loader.py” ties into ingestion by loading and batching the processed ECG segments.
ECG-Byte Tokenizer
• Training and Implementation: The “train_tokenizer.py” script (triggered via “train_tok.sh”) and the “rust_bpe” folder (compiled via maturin develop) are responsible for training the ECG-specific byte-pair encoding.
• Analysis of Tokenization: The analysis folder contains modules to plot token distribution (“token_dist.sh”) and track token merging (“track_encoding.sh”).
Model Training
There are two major training “pathways”:
A. End-to-End LLM Training:
• Main Training: “main.py” is the central script that ties together the loaded data, the preprocessed tokenized input, a chosen foundation model (e.g., Llama 3.2 1B) and the end-to-end architecture.
• Custom Scheduler: “scheduler.py” encapsulates a custom learning rate schedule used for both end-to-end and two-stage training.
• Runners: Scripts in the “runners” folder (e.g., inference.py, train.py, interpret.py) orchestrate the training and inferencing routines.
B. Two-Stage Training Methods:
• Stage 1 Pretraining: “pretrain.py” (invoked via “pretrain.sh”) processes and trains using models in the “models” folder (like a ResNet variant for feature extraction).
• Stage 2 Finetuning: “finetune.py” (run via “finetune.sh”) takes a checkpoint from stage 1 (passed as “first_check”) and fine-tunes further on the ECG-QA (question–answer) tasks.
Inference and Analysis
• Inference: Use “scripts/inference.sh” (which calls main.py with inference flag or “finetune.py” for two-stage) to generate answers from the trained LLM.
• Attention and Visualization: “interp_analysis.py” (and related scripts) provides visual analysis of attention maps. Also, the analysis folder has components to measure token distributions and merge tracking.
Auxiliary and Utility Modules
• Models: The “models” folder includes model definitions for the various network architectures. This abstraction is used in both training and finetuning.
• Utils: Contains helper functions across the codebase for file I/O, plotting (viz_utils.py), model checkpoint handling, and other repeated logic.
Execution Scripts & Environment
• “scripts” folder: Contains bash scripts for token training, end-to-end training, two-stage training, inference, and analysis.
Relationships and Data Flow:
• Dependencies like the custom scheduler feeding into training scripts.
Design Tips:
– Use clear labels for each component/block (e.g., “Raw Data”, “Preprocessing”, “ECG-Byte Tokenizer”, “End-to-End Training”, “Stage 1 Pretraining”, “Stage 2 Finetuning”, “Inference & Analysis”).
– Optionally, include annotations for scripts (e.g., “train_tok.sh”, “pretrain.sh”, “finetune.sh”) to show how the automation triggers various parts of the pipeline.
– Since many utilities and tests also exist (e.g. “test/” folder, “utils/”), you may omit extensive test details but note that the design supports thorough validation and modularity.
• Data Input (raw datasets, external mapping)
↓
• Preprocessing (data_loader, preprocess, sampling)
↓
• ECG-Byte Tokenizer (Rust BPE, train_tokenizer.py)
↓
• Training Pipelines:
• End-to-End LLM Training (main.py, model training, scheduler, runners)
• Two-Stage Training (pretrain.py → finetune.py; with checkpoint handoff)
↓
• Inference & Analysis (inference.sh, attention interp_analysis.py, token distribution visualizations)
• While running continuously on an environment set up with appropriate GPU/PyTorch and HuggingFace integration.