How nvidia/cosmos works
Cosmos is a model platform for physical-AI generation and reasoning; the supplied source excerpts chiefly expose its VLMEvalKit evaluation workflow. An evaluator selects datasets and models, runs inference, saves predictions, then scores them. The README also documents generator and reasoner use, while the tree includes generator evaluation suites. Core model-generation implementation is not shown, so those capabilities are represented only at the documented level; links without direct evidence are omitted.
Main entry points: run.py VLMEvalKit runner · __init__.py Dataset registry · inference.py Inference runners · config.py Model configuration
Components 22
Evaluation orchestration
- VLMEvalKit runner
evaluation/cosmos3/reasoner/vlmevalkit/run.py - Model configuration
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/config.py - Inference runners
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py - Video inference
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_video.py - API task pipeline
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_api.py - Result file utilities
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/smp/file.py
Benchmarks and scoring
- Dataset registry
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/__init__.py - Image benchmarks
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/image_vqa.py - CG benchmark
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/cgbench.py - Chart benchmark
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/chartmimic.py - Creation benchmark
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/creation.py - Answer extraction
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/utils/spatial_bench/llm_extract.py
Model interfaces
- Cosmos Reason API
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/cosmos_reason.py - LMDeploy API
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/lmdeploy.py - Spatial MLLM
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/vlm/spatial_mllm.py - LVS service adapter
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/lvs_service.py
Generator evaluation
- PAIBench scoring
evaluation/cosmos3/generator/paibench_g/run_motion_smoothness_sharded.py - UniGenBench scoring
evaluation/cosmos3/generator/unigenbench/ugb_scorer.py - UniGen utilities
evaluation/cosmos3/generator/unigenbench/utils.py - Cosmos score parsing
evaluation/cosmos3/reasoner/vlmevalkit/cosmos_eval/parse_score.py
Other components
- Evaluator
- Cosmos generator
Connections 17
- Evaluator → VLMEvalKit runner: starts evaluation
evaluation/cosmos3/reasoner/vlmevalkit/run.py - VLMEvalKit runner → Model configuration: loads configuration
evaluation/cosmos3/reasoner/vlmevalkit/run.py - VLMEvalKit runner → Dataset registry: selects datasets
evaluation/cosmos3/reasoner/vlmevalkit/run.py - VLMEvalKit runner → Inference runners: dispatches inference
evaluation/cosmos3/reasoner/vlmevalkit/run.py - VLMEvalKit runner → Video inference: dispatches video
evaluation/cosmos3/reasoner/vlmevalkit/run.py - Model configuration → Cosmos Reason API: configures API
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/config.py - Inference runners → Dataset registry: builds prompts
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py - Inference runners → Result file utilities: loads prior results
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py - Video inference → Result file utilities: resumes results
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_video.py - API task pipeline → Dataset registry: gets benchmark data
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_api.py - Dataset registry → Image benchmarks: registers benchmark
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/__init__.py - Image benchmarks → Answer extraction: uses extraction
evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/image_vqa.py - Cosmos generator → PAIBench scoring: evaluated by
README.md - Cosmos generator → UniGenBench scoring: evaluated by
- UniGenBench scoring → UniGen utilities: uses utilities
evaluation/cosmos3/generator/unigenbench/ugb_scorer.py - Evaluator → PAIBench scoring: runs evaluation
README.md - Evaluator → Cosmos score parsing: parses scores