# ciscoxu/embody-data-infrastructure architecture

> Architecture diagram of the GitHub repository ciscoxu/embody-data-infrastructure, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ciscoxu/embody-data-infrastructure
- Repository: https://github.com/ciscoxu/embody-data-infrastructure

## Overview

This is a local embodied-data infrastructure for acquiring raw recordings, inspecting and classifying sources, decoding them into vendor-neutral processed runs, and deriving separate purpose runs. The graph follows the documented four-layer workflow and the sampled processing orchestration; README-documented features with limited source excerpts are included at their exact tree paths. Detailed wiring for several unsampled modules is intentionally not inferred.

Main entry points: [`src/embody_data/processing.py`](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/processing.py) (Pipeline orchestration), [`src/embody_data/cli/main.py`](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/cli/main.py) (CLI commands), [`src/embody_data/acquisition/huggingface.py`](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/acquisition/huggingface.py) (Acquire files), [`src/embody_data/purpose/runner.py`](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/purpose/runner.py) (Purpose runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Raw acquisition"]
  node_acquire["Acquire files<br/>[huggingface.py]"]
  node_raw[("Immutable raw data")]
  node_remote["Remote file listing<br/>[huggingface.py]"]
end

subgraph group_discovery["Discovery and decoding"]
  node_inspect["Source inspection<br/>[mcap_inspect.py]"]
  node_inventory["Inventory and profile<br/>[source_profiles.py]"]
  node_profile["Source profile model<br/>[models.py]"]
  node_decode["GenRobot MCAP decode<br/>[genrobot.py]"]
end

subgraph group_processing["Semantic processing"]
  node_process["Pipeline orchestration<br/>[processing.py]"]
  node_representation["Normalized streams<br/>[genrobot.py]"]
  node_sync["Timeline alignment<br/>[alignment.py]"]
  node_calibration["Calibration validation<br/>[validation.py]"]
  node_episode["Episode derivation<br/>[recording.py]"]
  node_qc["QC and validity masks<br/>[report.py]"]
end

subgraph group_outputs["Runs and access"]
  node_cli["CLI commands<br/>[main.py]"]
  node_processed[("Processed run<br/>[layout.py]")]
  node_reader["Processed reader<br/>[reader.py]"]
  node_purpose["Purpose runner<br/>[runner.py]"]
  node_purposemodels["Purpose profiles and runs<br/>[models.py]"]
  node_purposeout[("Purpose run artifacts<br/>[purpose.py]")]
end

node_user(("Operator"))
node_huggingface["Hugging Face dataset"]

node_user -->|"runs commands"| node_cli
node_cli -->|"lists or downloads"| node_acquire
node_acquire -->|"queries and fetches"| node_huggingface
node_acquire -->|"writes raw files"| node_raw
node_acquire -->|"returns file manifest"| node_remote
node_cli -->|"inspects recording"| node_inspect
node_inspect -->|"reads recording"| node_raw
node_inspect -->|"builds inventory"| node_inventory
node_inventory -->|"uses profile model"| node_profile
node_cli -->|"processes recording"| node_process
node_process -->|"inspects source"| node_inspect
node_process -->|"resolves selected profile"| node_profile
node_process -->|"dispatches adapter"| node_decode
node_decode -->|"reads recording"| node_raw
node_decode -->|"writes normalized streams"| node_representation
node_process -->|"aligns timestamps"| node_sync
node_process -->|"validates calibration"| node_calibration
node_process -->|"derives episodes"| node_episode
node_process -->|"builds QC report"| node_qc
node_process -->|"writes processed run"| node_processed
node_cli -->|"describes or validates"| node_reader
node_reader -->|"reads processed run"| node_processed
node_cli -->|"creates purpose run"| node_purpose
node_purpose -->|"reads processed data"| node_reader
node_purpose -->|"uses run model"| node_purposemodels
node_purpose -->|"writes local artifacts"| node_purposeout

click node_cli "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/cli/main.py"
click node_acquire "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/acquisition/huggingface.py"
click node_inspect "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/ingestion/mcap_inspect.py"
click node_inventory "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/metadata/source_profiles.py"
click node_profile "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/metadata/models.py"
click node_process "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/processing.py"
click node_decode "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/decode/genrobot.py"
click node_representation "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/representation/genrobot.py"
click node_sync "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/sync/alignment.py"
click node_calibration "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/calibration/validation.py"
click node_episode "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/episode/recording.py"
click node_qc "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/qc/report.py"
click node_processed "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/storage/layout.py"
click node_reader "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/storage/reader.py"
click node_purpose "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/purpose/runner.py"
click node_purposemodels "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/purpose/models.py"
click node_purposeout "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/storage/purpose.py"
click node_remote "https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/acquisition/huggingface.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_acquire,node_raw,node_remote toneBlue
class node_inspect,node_inventory,node_profile,node_decode toneAmber
class node_process,node_representation,node_sync,node_calibration,node_episode,node_qc toneMint
class node_cli,node_processed,node_reader,node_purpose,node_purposemodels,node_purposeout toneRose
class node_user,node_huggingface toneIndigo
```

## Components

### Raw acquisition

- [Acquire files](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/acquisition/huggingface.py): `src/embody_data/acquisition/huggingface.py`
- **Immutable raw data**
- [Remote file listing](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/acquisition/huggingface.py): `src/embody_data/acquisition/huggingface.py`

### Discovery and decoding

- [Source inspection](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/ingestion/mcap_inspect.py): `src/embody_data/ingestion/mcap_inspect.py`
- [Inventory and profile](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/metadata/source_profiles.py): `src/embody_data/metadata/source_profiles.py`
- [Source profile model](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/metadata/models.py): `src/embody_data/metadata/models.py`
- [GenRobot MCAP decode](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/decode/genrobot.py): `src/embody_data/decode/genrobot.py`

### Semantic processing

- [Pipeline orchestration](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/processing.py): `src/embody_data/processing.py`
- [Normalized streams](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/representation/genrobot.py): `src/embody_data/representation/genrobot.py`
- [Timeline alignment](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/sync/alignment.py): `src/embody_data/sync/alignment.py`
- [Calibration validation](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/calibration/validation.py): `src/embody_data/calibration/validation.py`
- [Episode derivation](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/episode/recording.py): `src/embody_data/episode/recording.py`
- [QC and validity masks](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/qc/report.py): `src/embody_data/qc/report.py`

### Runs and access

- [CLI commands](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/cli/main.py): `src/embody_data/cli/main.py`
- [Processed run](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/storage/layout.py): `src/embody_data/storage/layout.py`
- [Processed reader](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/storage/reader.py): `src/embody_data/storage/reader.py`
- [Purpose runner](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/purpose/runner.py): `src/embody_data/purpose/runner.py`
- [Purpose profiles and runs](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/purpose/models.py): `src/embody_data/purpose/models.py`
- [Purpose run artifacts](https://github.com/ciscoxu/embody-data-infrastructure/blob/main/src/embody_data/storage/purpose.py): `src/embody_data/storage/purpose.py`

### Other components

- **Operator**
- **Hugging Face dataset**

## Connections

- Operator → CLI commands: runs commands
- CLI commands → Acquire files: lists or downloads
- Acquire files → Hugging Face dataset: queries and fetches
- Acquire files → Immutable raw data: writes raw files
- Acquire files → Remote file listing: returns file manifest
- CLI commands → Source inspection: inspects recording
- Source inspection → Immutable raw data: reads recording
- Source inspection → Inventory and profile: builds inventory
- Inventory and profile → Source profile model: uses profile model
- CLI commands → Pipeline orchestration: processes recording
- Pipeline orchestration → Source inspection: inspects source
- Pipeline orchestration → Source profile model: resolves selected profile
- Pipeline orchestration → GenRobot MCAP decode: dispatches adapter
- GenRobot MCAP decode → Immutable raw data: reads recording
- GenRobot MCAP decode → Normalized streams: writes normalized streams
- Pipeline orchestration → Timeline alignment: aligns timestamps
- Pipeline orchestration → Calibration validation: validates calibration
- Pipeline orchestration → Episode derivation: derives episodes
- Pipeline orchestration → QC and validity masks: builds QC report
- Pipeline orchestration → Processed run: writes processed run
- CLI commands → Processed reader: describes or validates
- Processed reader → Processed run: reads processed run
- CLI commands → Purpose runner: creates purpose run
- Purpose runner → Processed reader: reads processed data
- Purpose runner → Purpose profiles and runs: uses run model
- Purpose runner → Purpose run artifacts: writes local artifacts