How ciscoxu/embody-data-infrastructure works
This is a local embodied-data infrastructure for acquiring raw recordings, inspecting and classifying sources, decoding them into vendor-neutral processed runs, and deriving separate purpose runs. The graph follows the documented four-layer workflow and the sampled processing orchestration; README-documented features with limited source excerpts are included at their exact tree paths. Detailed wiring for several unsampled modules is intentionally not inferred.
Main entry points: processing.py Pipeline orchestration · main.py CLI commands · huggingface.py Acquire files · runner.py Purpose runner
Components 21
Raw acquisition
- Acquire files
src/embody_data/acquisition/huggingface.py - Immutable raw data
- Remote file listing
src/embody_data/acquisition/huggingface.py
Discovery and decoding
- Source inspection
src/embody_data/ingestion/mcap_inspect.py - Inventory and profile
src/embody_data/metadata/source_profiles.py - Source profile model
src/embody_data/metadata/models.py - GenRobot MCAP decode
src/embody_data/decode/genrobot.py
Semantic processing
- Pipeline orchestration
src/embody_data/processing.py - Normalized streams
src/embody_data/representation/genrobot.py - Timeline alignment
src/embody_data/sync/alignment.py - Calibration validation
src/embody_data/calibration/validation.py - Episode derivation
src/embody_data/episode/recording.py - QC and validity masks
src/embody_data/qc/report.py
Runs and access
- CLI commands
src/embody_data/cli/main.py - Processed run
src/embody_data/storage/layout.py - Processed reader
src/embody_data/storage/reader.py - Purpose runner
src/embody_data/purpose/runner.py - Purpose profiles and runs
src/embody_data/purpose/models.py - Purpose run artifacts
src/embody_data/storage/purpose.py
Other components
- Operator
- Hugging Face dataset
Connections 26
- Operator → CLI commands: runs commands
- CLI commands → Acquire files: lists or downloads
- Acquire files → Hugging Face dataset: queries and fetches
- Acquire files → Immutable raw data: writes raw files
- Acquire files → Remote file listing: returns file manifest
- CLI commands → Source inspection: inspects recording
- Source inspection → Immutable raw data: reads recording
- Source inspection → Inventory and profile: builds inventory
- Inventory and profile → Source profile model: uses profile model
- CLI commands → Pipeline orchestration: processes recording
- Pipeline orchestration → Source inspection: inspects source
- Pipeline orchestration → Source profile model: resolves selected profile
- Pipeline orchestration → GenRobot MCAP decode: dispatches adapter
- GenRobot MCAP decode → Immutable raw data: reads recording
- GenRobot MCAP decode → Normalized streams: writes normalized streams
- Pipeline orchestration → Timeline alignment: aligns timestamps
- Pipeline orchestration → Calibration validation: validates calibration
- Pipeline orchestration → Episode derivation: derives episodes
- Pipeline orchestration → QC and validity masks: builds QC report
- Pipeline orchestration → Processed run: writes processed run
- CLI commands → Processed reader: describes or validates
- Processed reader → Processed run: reads processed run
- CLI commands → Purpose runner: creates purpose run
- Purpose runner → Processed reader: reads processed data
- Purpose runner → Purpose profiles and runs: uses run model
- Purpose runner → Purpose run artifacts: writes local artifacts