How nvidia/physicsnemo-curator works
PhysicsNeMo Curator is an ETL toolkit for turning scientific inputs into AI-ready mesh, weather/data-array, and atomic datasets. Users either assemble a fluent Source → Filter → Sink pipeline or use the Textual wizard, then run it through a selectable execution backend. PipelineStore records configuration, metrics, errors, and artifacts; the Panel dashboard reads that database for overview, pipeline, and performance inspection. The tree and excerpts support these major capabilities, though several stage-to-stage implementations were not sampled.
Components 20
User Interfaces
- Interactive Wizard
src/physicsnemo_curator/wiz/app.py
Pipeline Core
- Component Registry
src/physicsnemo_curator/core/registry.py - Pipeline API
src/physicsnemo_curator/core/base.py
Domain Processing
- Mesh Sources
src/physicsnemo_curator/domains/mesh/sources - Mesh Filters
src/physicsnemo_curator/domains/mesh/filters - Mesh Sinks
src/physicsnemo_curator/domains/mesh/sinks - DataArray Sources
src/physicsnemo_curator/domains/da/sources - DataArray Filters
src/physicsnemo_curator/domains/da/filters - DataArray Sinks
src/physicsnemo_curator/domains/da/sinks - Atomic Sources
src/physicsnemo_curator/domains/atm/sources - Atomic Filters
src/physicsnemo_curator/domains/atm/filters - Atomic Sinks
src/physicsnemo_curator/domains/atm/sinks - Native Readers
src/rust/src/lib.rs
Execution
- Pipeline Runner
src/physicsnemo_curator/run/base.py - Execution Backends
src/physicsnemo_curator/run
Observability
- Pipeline Store
src/physicsnemo_curator/core/pipeline_store.py - Metrics Dashboard
src/physicsnemo_curator/dashboard/app.py
Other components
- Pipeline User
- Scientific Inputs
- Curated Datasets
Connections 35
- Pipeline User → Interactive Wizard: configures
- Pipeline User → Pipeline API: builds pipeline
- Interactive Wizard → Component Registry: discovers components
- Component Registry → Mesh Sources: resolves plugins
- Component Registry → Mesh Filters: resolves plugins
- Component Registry → Mesh Sinks: resolves plugins
- Component Registry → DataArray Sources: resolves plugins
- Component Registry → DataArray Filters: resolves plugins
- Component Registry → DataArray Sinks: resolves plugins
- Component Registry → Atomic Sources: resolves plugins
- Component Registry → Atomic Filters: resolves plugins
- Component Registry → Atomic Sinks: resolves plugins
- Interactive Wizard → Pipeline API: assembles pipeline
- Pipeline API → Pipeline Runner: accepts execution
- Pipeline Runner → Execution Backends: dispatches work
- Pipeline Runner → Pipeline Store: records metrics
- Pipeline Runner → Mesh Sources: pulls items
- Pipeline Runner → Mesh Filters: applies filters
- Pipeline Runner → Mesh Sinks: dispatches sink
- Pipeline Runner → DataArray Sources: pulls items
- Pipeline Runner → DataArray Filters: applies filters
- Pipeline Runner → DataArray Sinks: dispatches sink
- Pipeline Runner → Atomic Sources: pulls items
- Pipeline Runner → Atomic Filters: applies filters
- Pipeline Runner → Atomic Sinks: dispatches sink
- Scientific Inputs → Mesh Sources: read inputs
- Scientific Inputs → DataArray Sources: read inputs
- Scientific Inputs → Atomic Sources: read inputs
- Mesh Sources → Native Readers: uses parsers
- Atomic Sources → Native Readers: uses parsers
- Mesh Sinks → Curated Datasets: writes datasets
- DataArray Sinks → Curated Datasets: writes datasets
- Atomic Sinks → Curated Datasets: writes datasets
- Metrics Dashboard → Pipeline Store: reads runs
- Pipeline User → Metrics Dashboard: inspects results