How arkel23/lalmdecoderanalysis works
Purpose and Scope
LALMDecoderAnalysis is an analysis-only research repository for comparing LisTAya/TinyAya decoder-SFT variants in a listening-language-model setup. It does not train models or run evaluations; it ingests experiment metadata and histories from Weights & Biases, derives statistical results, plots, and generates the ACL paper artifacts. Primary orientation and conclusions live in README.md, docs/FINDINGS.md, and docs/OUTPUTS.md.
Primary Entry Points and Orchestration
The end-to-end rebuild entry point is plotter.sh, intended for the pytorch environment. It coordinates data accounting, analysis, plotting, paper-table generation, and verification. Dataset-dependent checks require the separate asr environment. The publication endpoint is ACL26_LALMDecoder/main.tex; its generated tables reside under ACL26_LALMDecoder/tables/.
Experiment Ingestion Boundary
W&B is the central external service and authoritative source for run-level metrics. download_save_wandb_data.py downloads one experiment serial at a time into normalized raw CSVs such as data/raw_serials/raw_serial_0.csv; download_wandb_history.py retrieves step-level histories. Serial IDs partition experiment cohorts: serial 0 is the canonical 48-run grid, while later serials contain controls, reruns, and baseline/checkpoint work. rename_wandb_serial.py and missing_runs.py support serial management and sweep-gap detection.
Canonical Data and Preprocessing
Downloaded data are retained as CSV artifacts, then transformed through shared experiment dictionaries and preprocessing logic in utils.py. concat_df.py and count_data.py establish combined/accounted run data before downstream claims are calculated. Static manifests and supporting composition evidence form a separate input boundary: data/manifest_training.csv, data/manifest_eval_sets.csv, and data/tinyaya_report/tinyaya_language_composition.csv.
Analysis Stages
The analytical core is a family of focused analyze_*.py scripts that derive the paper’s t1–t9 result sets: sample efficiency, regional match, OOD/CRS, data accounting, volume interactions, loss metrics, baselines, exposure, and replicate effects. Representative stages are analyze_sample_efficiency.py, analyze_loss_metrics.py, and analyze_baselines.py. Outputs are persisted as reproducible CSVs in results_all/acc/, including results_all/acc/t1_sample_efficiency.csv and results_all/acc/t6_loss_metrics.csv, rather than being computed inside the TeX document.
Visualization and Publication Generation
plot_curve.py is the primary figure renderer, consuming accumulated analysis data and histories to write plot artifacts under results_all/plots/. build_paper_tables.py translates derived result CSVs into TeX table fragments consumed by the paper, including ACL26_LALMDecoder/tables/tab_baselines.tex and ACL26_LALMDecoder/tables/tab_loss_by_tier.tex. This keeps research calculations, presentation tables, and the manuscript (ACL26_LALMDecoder/main.tex) as distinct layers.
Validation and Upstream Boundary
Verification scripts are terminal pipeline guards: verify_paper_numbers.py, verify_dataset_durations.py, verify_eval_pairing.py, and verify_interleave_semantics.py fail on inconsistencies between source data, derived outputs, and reported claims. for_quantizedasr/ contains evaluation/sweep YAML generators and shell launchers intended for the separate QuantizedASR repository, such as for_quantizedasr/tools/preprocess/create_yamls_models_lalm_txf.py; this repository reviews and prepares upstream work but does not modify or execute that training/evaluation system.
Components 18
Inputs
External experiment and composition sources.
- Weights & Biases · Authoritative run-level metrics and histories.
- Dataset manifests · Training, evaluation, and language-composition inputs.
data/manifest_training.csv
Analysis Pipeline
Reproducible data, analysis, and rendering stages.
- Serial downloader · Downloads one experiment serial into normalized raw CSVs.
download_save_wandb_data.py - History downloader · Retrieves step-level W&B histories.
download_wandb_history.py - Raw serial CSVs · Serial 0 is the canonical 48-run grid; later serials cover controls and reruns.
data/raw_serials/raw_serial_0.csv - Serial management · Renames serials and detects missing sweep runs.
rename_wandb_serial.py - Experiment preprocessing · Shared dictionaries and preprocessing normalize experiment data.
utils.py - Run accounting · Combines and accounts for downloaded runs.
concat_df.py - Result analyses t1–t9 · Focused analyses derive the paper result sets.
analyze_sample_efficiency.py - Derived result CSVs · Persisted statistical outputs used by plots and tables.
results_all/acc/t1_sample_efficiency.csv - Figure renderer · Renders figures from accumulated analyses and histories.
plot_curve.py - Plot artifacts
- End-to-end rebuild · Coordinates accounting, analysis, plotting, table generation, and verification.
plotter.sh
Publication
Generated paper artifacts and manuscript.
- Paper table builder · Translates derived CSVs into TeX table fragments.
build_paper_tables.py - Generated TeX tables
- ACL manuscript · Publication endpoint consuming generated tables.
ACL26_LALMDecoder/main.tex
Validation & Upstream
Pipeline guards and external-system preparation.
- Consistency verification · Fails when source data, derived outputs, and reported claims disagree.
verify_paper_numbers.py - QuantizedASR preparation · Generates evaluation and sweep inputs for the separate QuantizedASR repository.
for_quantizedasr/tools/preprocess/create_yamls_models_lalm_txf.py
Connections 24
- Weights & Biases → Serial downloader: run metadata
- Weights & Biases → History downloader: step histories
- Serial downloader → Raw serial CSVs: writes
- Serial management → Raw serial CSVs: manages cohorts
- Raw serial CSVs → Experiment preprocessing: normalizes
- Dataset manifests → Experiment preprocessing: composition context
- Experiment preprocessing → Run accounting: prepared runs
- Run accounting → Result analyses t1–t9: accounted data
- Dataset manifests → Result analyses t1–t9: dataset evidence
- Result analyses t1–t9 → Derived result CSVs: writes t1–t9
- Derived result CSVs → Figure renderer: analysis data
- History downloader → Figure renderer: histories
- Figure renderer → Plot artifacts: renders
- Derived result CSVs → Paper table builder: derived claims
- Paper table builder → Generated TeX tables: generates
- Generated TeX tables → ACL manuscript: included by
- End-to-end rebuild → Run accounting: runs
- End-to-end rebuild → Result analyses t1–t9: runs
- End-to-end rebuild → Figure renderer: runs
- End-to-end rebuild → Paper table builder: runs
- End-to-end rebuild → Consistency verification: runs guards
- Consistency verification → Derived result CSVs: checks
- Consistency verification → ACL manuscript: checks claims
- QuantizedASR preparation → Dataset manifests: prepares external workflow