# treqs/roar architecture

> Architecture diagram of the GitHub repository treqs/roar, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/treqs/roar
- Repository: https://github.com/treqs/roar

## Overview

1. **What this repo is**
   Roar is a provenance and artifact-registration system for ML/data workflows. Its core architectural unit is a content-addressed DAG of sessions, jobs, and artifacts stored locally, with optional publication to GLaaS and cloud storage.

2. **Top-level layering**
   The Python package is organized as a layered system: `roar/cli` handles user commands, `roar/application` contains use-case orchestration, `roar/core` defines shared domain models and interfaces, `roar/db` owns persistence, and `roar/integrations` wraps external systems. `roar/presenters` is the output/rendering layer for terminal and TUI views.

3. **Local state and persistence**
   The local source of truth is a SQLite database under `.roar/roar.db`, initialized by `roar init`. The DB stores artifact hashes, job records, sessions, labels, collections, and a hash cache, which are then queried by commands like `dag`, `show`, `status`, `diff`, `log`, and `lineage`.

4. **Run/observe pipeline**
   `roar run` is the main capture path: it launches a command, observes file I/O and execution metadata, and records the resulting job plus inputs/outputs into the local session graph. The runtime layer (`roar/execution/runtime`) handles bootstrap, injection, tracing lifecycle, signal handling, and backend selection.

5. **Native tracing subsystem**
   Observation is implemented by separate Rust tracer binaries, not by Python alone. The Rust workspace contains multiple backend families: eBPF, preload, and ptrace tracers, plus a proxy service and helper crates; the Python package stages these binaries into `roar/bin` for runtime discovery.

6. **Backend selection and host execution**
   The architecture treats tracing as pluggable infrastructure. `roar/backends/ray`, `roar/backends/osmo`, and `roar/backends/local` adapt Roar to different execution environments, while the runtime and backend config determine how jobs are submitted, attached, or proxied.

7. **Publishing and registration flow**
   `roar register` and `roar put` turn local lineage into a GLaaS registration package, optionally uploading artifacts to cloud storage. The publish application layer assembles lineage, source resolution, job metadata, labels, and visibility/auth context before calling the GLaaS client and storage backends.

8. **Artifact acquisition and reproducibility**
   `roar get` downloads artifacts from S3/GCS/HF/HTTP sources and records them as source nodes in the local DAG. `roar reproduce` uses lineage lookup plus environment setup and pipeline execution helpers to reconstruct the steps required to rebuild a target artifact or full session.

9. **Query and visualization boundary**
   Read-only commands are separated from mutation/publish commands. Query services build DAGs, diffs, labels, and lineage views from the DB and current session state, while presenters render those structures in console output or the TUI screens.

10. **External integrations**
   The repo integrates with Git for commit/branch capture and repository context, with cloud/object stores for artifact transfer, with GLaaS for global lineage publication, and with telemetry for coarse usage counters. These integrations are isolated behind dedicated adapters so core lineage logic stays environment-agnostic.

11. **Auth, scope, and labels**
   Authentication and publication scope are first-class boundaries: `auth`/`login` manage GLaaS identity, `scope` determines where lineage is published, and `label` manages local metadata that can later be synchronized. This keeps user identity and project ownership separate from the provenance graph itself.

12. **Build and packaging boundary**
   The project is a Python package plus Rust workspace, with GitHub Actions building and publishing wheels that include native binaries. Editable/source installs need the Rust artifacts built separately, which is why the repo includes staging scripts and binary-copy actions.

13. **What to diagram**
   The most important diagram is a left-to-right flow from CLI command → application service → core model/repository → DB and integrations → optional publish/download/reproduce endpoints, with the Rust tracer/runtime path drawn underneath `run`. A second boundary should show the local system versus external services: filesystem, Git, cloud storage, telemetry, and GLaaS.

Main entry points: [`roar/execution/runtime`](https://github.com/treqs/roar/tree/main/roar/execution/runtime) (Runtime), [`roar/application/publish/service.py`](https://github.com/treqs/roar/blob/main/roar/application/publish/service.py) (Publish service), [`roar/db`](https://github.com/treqs/roar/tree/main/roar/db) (SQLite DB), [`roar/application/run/service.py`](https://github.com/treqs/roar/blob/main/roar/application/run/service.py) (Run service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["CLI"]
  node_roar_cli["CLI entry<br/>entrypoint"]
  node_roar_cli_run["Run cmd<br/>command<br/>[run.py]"]
  node_roar_cli_query["Query cmds<br/>command-surface<br/>[dag.py]"]
  node_roar_cli_publish["Publish cmds<br/>command-surface<br/>[register.py]"]
end

subgraph group_app["Application"]
  node_roar_app_run["Run service<br/>use-case<br/>[service.py]"]
  node_roar_app_query["Query service<br/>use-case<br/>[dag.py]"]
  node_roar_app_get["Get service<br/>use-case<br/>[service.py]"]
  node_roar_app_publish["Publish service<br/>use-case<br/>[service.py]"]
  node_roar_app_reproduce["Reproduce svc<br/>use-case<br/>[service.py]"]
  node_roar_app_workflow["Workflow svc<br/>orchestration<br/>[service.py]"]
end

subgraph group_core["Core"]
  node_roar_core[("Domain core")]
end

subgraph group_db["Persistence"]
  node_roar_db[("SQLite DB<br/>persistence")]
end

subgraph group_exec["Execution"]
  node_roar_exec_runtime["Runtime"]
  node_roar_exec_provenance["Provenance<br/>capture-pipeline<br/>[service.py]"]
  node_roar_exec_reproduction["Rebuild path<br/>reproduction-pipeline"]
end

subgraph group_integrations["Integrations"]
  node_roar_integrations_git["Git adapter<br/>[context.py]"]
  node_roar_integrations_storage["Storage<br/>adapter<br/>[publish.py]"]
  node_roar_integrations_glaas["GLaaS client<br/>adapter<br/>[client.py]"]
end

subgraph group_ui["Views"]
  node_roar_presenters["Presenters<br/>rendering<br/>[dag_renderer.py]"]
  node_roar_tui["TUI<br/>[app.py]"]
end

subgraph group_rust["Rust"]
  node_rust_runtime["Tracer runtime<br/>native-runtime<br/>[lib.rs]"]
  node_rust_ebpf["eBPF tracer<br/>native-tracer<br/>[main.rs]"]
  node_rust_preload["Preload tracer<br/>native-tracer<br/>[main.rs]"]
  node_rust_ptrace["Ptrace tracer<br/>native-tracer<br/>[main.rs]"]
  node_rust_proxy["Proxy svc<br/>native-service<br/>[main.rs]"]
end

node_ext_filesystem[("Filesystem<br/>external-system")]
node_ext_git["Git host<br/>external-system"]
node_ext_cloud[("Cloud storage<br/>external-system")]
node_ext_glaas["GLaaS<br/>external-service"]
node_ext_telemetry(("Telemetry<br/>external-service"))

node_roar_cli -->|"dispatches"| node_roar_cli_run
node_roar_cli -->|"dispatches"| node_roar_cli_query
node_roar_cli -->|"dispatches"| node_roar_cli_publish
node_roar_cli_run -->|"calls"| node_roar_app_run
node_roar_cli_query -->|"calls"| node_roar_app_query
node_roar_cli_publish -->|"calls"| node_roar_app_publish
node_roar_app_run -->|"bootstraps"| node_roar_exec_runtime
node_roar_exec_runtime -->|"loads"| node_rust_runtime
node_roar_exec_runtime -->|"selects"| node_rust_ebpf
node_roar_exec_runtime -->|"selects"| node_rust_preload
node_roar_exec_runtime -->|"selects"| node_rust_ptrace
node_roar_exec_runtime -->|"feeds"| node_roar_exec_provenance
node_roar_exec_provenance -->|"records"| node_roar_db
node_roar_app_run -->|"reads/writes"| node_roar_db
node_roar_app_query -->|"reads"| node_roar_db
node_roar_app_get -->|"downloads"| node_roar_integrations_storage
node_roar_app_get -->|"records"| node_roar_db
node_roar_app_publish -->|"publishes"| node_roar_integrations_glaas
node_roar_app_publish -->|"uploads"| node_roar_integrations_storage
node_roar_app_publish -->|"uses"| node_roar_integrations_git
node_roar_app_reproduce -->|"looks up"| node_roar_app_query
node_roar_app_reproduce -->|"drives"| node_roar_exec_reproduction
node_roar_app_workflow -->|"models"| node_roar_core
node_roar_presenters -->|"renders"| node_roar_app_query
node_roar_presenters -->|"formats"| node_roar_db
node_roar_tui -->|"uses"| node_roar_presenters
node_roar_app_publish -->|"maps"| node_roar_core
node_roar_app_run -->|"depends on"| node_roar_core
node_roar_db -->|"implements"| node_roar_core
node_roar_integrations_glaas -->|"connects"| node_ext_glaas
node_roar_integrations_storage -->|"connects"| node_ext_cloud
node_roar_integrations_git -->|"connects"| node_ext_git
node_roar_presenters -->|"reads"| node_ext_filesystem
node_roar_exec_runtime -->|"observes"| node_ext_filesystem
node_rust_proxy -->|"relays"| node_ext_cloud
node_roar_app_publish -.->|"reports"| node_ext_telemetry

click node_roar_cli "https://github.com/treqs/roar/blob/main/roar/cli/command_registry.py"
click node_roar_cli_run "https://github.com/treqs/roar/blob/main/roar/cli/commands/run.py"
click node_roar_cli_query "https://github.com/treqs/roar/blob/main/roar/cli/commands/dag.py"
click node_roar_cli_publish "https://github.com/treqs/roar/blob/main/roar/cli/commands/register.py"
click node_roar_app_run "https://github.com/treqs/roar/blob/main/roar/application/run/service.py"
click node_roar_app_query "https://github.com/treqs/roar/blob/main/roar/application/query/dag.py"
click node_roar_app_get "https://github.com/treqs/roar/blob/main/roar/application/get/service.py"
click node_roar_app_publish "https://github.com/treqs/roar/blob/main/roar/application/publish/service.py"
click node_roar_app_reproduce "https://github.com/treqs/roar/blob/main/roar/application/reproduce/service.py"
click node_roar_app_workflow "https://github.com/treqs/roar/blob/main/roar/application/workflow/service.py"
click node_roar_core "https://github.com/treqs/roar/tree/main/roar/core"
click node_roar_db "https://github.com/treqs/roar/tree/main/roar/db"
click node_roar_exec_runtime "https://github.com/treqs/roar/tree/main/roar/execution/runtime"
click node_roar_exec_provenance "https://github.com/treqs/roar/blob/main/roar/execution/provenance/service.py"
click node_roar_exec_reproduction "https://github.com/treqs/roar/blob/main/roar/execution/reproduction/pipeline_executor.py"
click node_roar_integrations_git "https://github.com/treqs/roar/blob/main/roar/integrations/git/context.py"
click node_roar_integrations_storage "https://github.com/treqs/roar/blob/main/roar/integrations/storage/publish.py"
click node_roar_integrations_glaas "https://github.com/treqs/roar/blob/main/roar/integrations/glaas/client.py"
click node_roar_presenters "https://github.com/treqs/roar/blob/main/roar/presenters/dag_renderer.py"
click node_roar_tui "https://github.com/treqs/roar/blob/main/roar/tui/app.py"
click node_rust_runtime "https://github.com/treqs/roar/blob/main/rust/crates/tracer-runtime/src/lib.rs"
click node_rust_ebpf "https://github.com/treqs/roar/blob/main/rust/tracers/ebpf/userspace/src/main.rs"
click node_rust_preload "https://github.com/treqs/roar/blob/main/rust/tracers/preload/src/main.rs"
click node_rust_ptrace "https://github.com/treqs/roar/blob/main/rust/tracers/ptrace/src/main.rs"
click node_rust_proxy "https://github.com/treqs/roar/blob/main/rust/services/proxy/src/main.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_roar_cli,node_roar_cli_run,node_roar_cli_query,node_roar_cli_publish,node_roar_presenters,node_roar_tui toneBlue
class node_roar_app_run,node_roar_app_query,node_roar_app_get,node_roar_app_publish,node_roar_app_reproduce,node_roar_app_workflow,node_rust_runtime,node_rust_ebpf,node_rust_preload,node_rust_ptrace,node_rust_proxy toneAmber
class node_roar_core toneMint
class node_roar_db toneRose
class node_roar_exec_runtime,node_roar_exec_provenance,node_roar_exec_reproduction toneIndigo
class node_roar_integrations_git,node_roar_integrations_storage,node_roar_integrations_glaas toneTeal
class node_ext_filesystem,node_ext_git,node_ext_cloud,node_ext_glaas,node_ext_telemetry toneNeutral
```

## Components

### CLI

User commands and entrypoints

- [CLI entry](https://github.com/treqs/roar/blob/main/roar/cli/command_registry.py): entrypoint · Command registration and top-level CLI wiring · `roar/cli/command_registry.py`
- [Run cmd](https://github.com/treqs/roar/blob/main/roar/cli/commands/run.py): command · Launches and captures a command execution · `roar/cli/commands/run.py`
- [Query cmds](https://github.com/treqs/roar/blob/main/roar/cli/commands/dag.py): command-surface · Read-only DAG, status, show, diff, lineage, and log commands · `roar/cli/commands/dag.py`
- [Publish cmds](https://github.com/treqs/roar/blob/main/roar/cli/commands/register.py): command-surface · register, put, scope, auth, login, and related publishing commands · `roar/cli/commands/register.py`

### Application

Use-case orchestration layer

- [Run service](https://github.com/treqs/roar/blob/main/roar/application/run/service.py): use-case · Orchestrates capture, bootstrap, and session recording · `roar/application/run/service.py`
- [Query service](https://github.com/treqs/roar/blob/main/roar/application/query/dag.py): use-case · Builds DAGs, diffs, status, show, and lineage views from local state · `roar/application/query/dag.py`
- [Get service](https://github.com/treqs/roar/blob/main/roar/application/get/service.py): use-case · Resolves and downloads remote artifacts into local DAG state · `roar/application/get/service.py`
- [Publish service](https://github.com/treqs/roar/blob/main/roar/application/publish/service.py): use-case · Prepares registration packages and publishes lineage to GLaaS · `roar/application/publish/service.py`
- [Reproduce svc](https://github.com/treqs/roar/blob/main/roar/application/reproduce/service.py): use-case · Looks up lineage and reconstructs execution environment and steps · `roar/application/reproduce/service.py`
- [Workflow svc](https://github.com/treqs/roar/blob/main/roar/application/workflow/service.py): orchestration · Generates or coordinates workflow definitions and execution plans · `roar/application/workflow/service.py`

### Core

Domain models and interfaces

- [Domain core](https://github.com/treqs/roar/tree/main/roar/core): domain · Shared models, interfaces, validation, and provenance concepts · `roar/core`

### Persistence

SQLite storage and repositories

- [SQLite DB](https://github.com/treqs/roar/tree/main/roar/db): persistence · Database context, schema, and repository layer under .roar/roar.db · `roar/db`

### Execution

Run-time capture and reproduction

- [Runtime](https://github.com/treqs/roar/tree/main/roar/execution/runtime): runtime · Injection, tracing lifecycle, bootstrap, signals, and backend selection · `roar/execution/runtime`
- [Provenance](https://github.com/treqs/roar/blob/main/roar/execution/provenance/service.py): capture-pipeline · Collects file, process, package, and dataset metadata during runs · `roar/execution/provenance/service.py`
- [Rebuild path](https://github.com/treqs/roar/blob/main/roar/execution/reproduction/pipeline_executor.py): reproduction-pipeline · Environment setup and pipeline execution helpers for reproduction · `roar/execution/reproduction/pipeline_executor.py`

### Integrations

Adapters to external systems

- [Git adapter](https://github.com/treqs/roar/blob/main/roar/integrations/git/context.py): adapter · Repository context, commit, branch, and readiness discovery · `roar/integrations/git/context.py`
- [Storage](https://github.com/treqs/roar/blob/main/roar/integrations/storage/publish.py): adapter · Cloud and memory storage backends for upload and download · `roar/integrations/storage/publish.py`
- [GLaaS client](https://github.com/treqs/roar/blob/main/roar/integrations/glaas/client.py): adapter · Auth, transport, and registration client for publication · `roar/integrations/glaas/client.py`

### Views

Console and TUI rendering

- [Presenters](https://github.com/treqs/roar/blob/main/roar/presenters/dag_renderer.py): rendering · Console rendering, DAG formatting, diffs, and run reports · `roar/presenters/dag_renderer.py`
- [TUI](https://github.com/treqs/roar/blob/main/roar/tui/app.py): ui · Interactive terminal screens and widgets · `roar/tui/app.py`

### Rust

Native tracers and helper services

- [Tracer runtime](https://github.com/treqs/roar/blob/main/rust/crates/tracer-runtime/src/lib.rs): native-runtime · Native runtime support and discovery for tracer binaries · `rust/crates/tracer-runtime/src/lib.rs`
- [eBPF tracer](https://github.com/treqs/roar/blob/main/rust/tracers/ebpf/userspace/src/main.rs): native-tracer · eBPF probe and userspace daemon for syscall/file observation · `rust/tracers/ebpf/userspace/src/main.rs`
- [Preload tracer](https://github.com/treqs/roar/blob/main/rust/tracers/preload/src/main.rs): native-tracer · LD_PRELOAD-based interposition tracer and helpers · `rust/tracers/preload/src/main.rs`
- [Ptrace tracer](https://github.com/treqs/roar/blob/main/rust/tracers/ptrace/src/main.rs): native-tracer · Ptrace-based tracer backend · `rust/tracers/ptrace/src/main.rs`
- [Proxy svc](https://github.com/treqs/roar/blob/main/rust/services/proxy/src/main.rs): native-service · Native proxy service for forwarding and S3 relay · `rust/services/proxy/src/main.rs`

### Other components

- **Filesystem**: external-system · Local working tree, session graph, and captured artifacts
- **Git host**: external-system · Repository metadata and remote origins
- **Cloud storage**: external-system · S3, GCS, and Hugging Face artifact sources and sinks
- **GLaaS**: external-service · Global lineage registration and lookup service
- **Telemetry**: external-service · Usage and product telemetry destination

## Connections

- CLI entry → Run cmd: dispatches · Registers and routes commands
- CLI entry → Query cmds: dispatches · Routes query subcommands
- CLI entry → Publish cmds: dispatches · Routes auth and publish subcommands
- Run cmd → Run service: calls · Invokes the run orchestration
- Query cmds → Query service: calls · Invokes read-only query services
- Publish cmds → Publish service: calls · Invokes registration and put flows
- Run service → Runtime: bootstraps · Starts injected runtime and tracer lifecycle
- Runtime → Tracer runtime: loads · Selects and launches native tracing runtime
- Runtime → eBPF tracer: selects · Uses eBPF backend when available
- Runtime → Preload tracer: selects · Uses preload backend when available
- Runtime → Ptrace tracer: selects · Uses ptrace backend when available
- Runtime → Provenance: feeds · Passes tracing events into provenance capture
- Provenance → SQLite DB: records · Stores jobs, sessions, artifacts, and hashes locally
- Run service → SQLite DB: reads/writes · Updates local session graph and job records
- Query service → SQLite DB: reads · Builds DAG, diff, status, and lineage views
- Get service → Storage: downloads · Pulls artifacts from remote object/document sources
- Get service → SQLite DB: records · Registers downloaded artifacts as source nodes
- Publish service → GLaaS client: publishes · Sends registration packages and lineage payloads
- Publish service → Storage: uploads · Optionally stages artifacts to cloud storage
- Publish service → Git adapter: uses · Captures commit, branch, and remote context
- Reproduce svc → Query service: looks up · Reuses lineage and DAG query results
- Reproduce svc → Rebuild path: drives · Runs environment setup and execution reconstruction
- Workflow svc → Domain core: models · Builds workflow requests and results against core contracts
- Presenters → Query service: renders · Converts query results to terminal output
- Presenters → SQLite DB: formats · Consumes stored DAG and diff structures
- TUI → Presenters: uses · Interactive screens reuse presenter formatting and data
- Publish service → Domain core: maps · Assembles domain models, labels, and registration DTOs
- Run service → Domain core: depends on · Uses shared provenance and run domain contracts
- SQLite DB → Domain core: implements · Repository and service interfaces are defined in core
- GLaaS client → GLaaS: connects · Publishes to the external GLaaS service
- Storage → Cloud storage: connects · Uploads and downloads from cloud/object stores
- Git adapter → Git host: connects · Reads repository metadata from Git
- Presenters → Filesystem: reads · Displays local DAG and run state
- Runtime → Filesystem: observes · Captures file I/O and execution effects
- Proxy svc → Cloud storage: relays · Forwards traffic and S3 access for proxy modes
- Publish service → Telemetry: reports · May emit coarse usage telemetry during publish flows