treqs/roar

1. What this repo is

Roar is a provenance and artifact-registration system for ML/data workflows. Its core architectural unit is a content-addressed DAG of sessions, jobs, and artifacts stored locally, with optional publication to GLaaS and cloud storage.

2. Top-level layering

The Python package is organized as a layered system: roar/cli handles user commands, roar/application contains use-case orchestration, roar/core defines shared domain models and interfaces, roar/db owns persistence, and roar/integrations wraps external systems. roar/presenters is the output/rendering layer for terminal and TUI views.

3. Local state and persistence

The local source of truth is a SQLite database under .roar/roar.db, initialized by roar init. The DB stores artifact hashes, job records, sessions, labels, collections, and a hash cache, which are then queried by commands like dag, show, status, diff, log, and lineage.

4. Run/observe pipeline

roar run is the main capture path: it launches a command, observes file I/O and execution metadata, and records the resulting job plus inputs/outputs into the local session graph. The runtime layer (roar/execution/runtime) handles bootstrap, injection, tracing lifecycle, signal handling, and backend selection.

5. Native tracing subsystem

Observation is implemented by separate Rust tracer binaries, not by Python alone. The Rust workspace contains multiple backend families: eBPF, preload, and ptrace tracers, plus a proxy service and helper crates; the Python package stages these binaries into roar/bin for runtime discovery.

6. Backend selection and host execution

The architecture treats tracing as pluggable infrastructure. roar/backends/ray, roar/backends/osmo, and roar/backends/local adapt Roar to different execution environments, while the runtime and backend config determine how jobs are submitted, attached, or proxied.

7. Publishing and registration flow

roar register and roar put turn local lineage into a GLaaS registration package, optionally uploading artifacts to cloud storage. The publish application layer assembles lineage, source resolution, job metadata, labels, and visibility/auth context before calling the GLaaS client and storage backends.

8. Artifact acquisition and reproducibility

roar get downloads artifacts from S3/GCS/HF/HTTP sources and records them as source nodes in the local DAG. roar reproduce uses lineage lookup plus environment setup and pipeline execution helpers to reconstruct the steps required to rebuild a target artifact or full session.

9. Query and visualization boundary

Read-only commands are separated from mutation/publish commands. Query services build DAGs, diffs, labels, and lineage views from the DB and current session state, while presenters render those structures in console output or the TUI screens.

10. External integrations

The repo integrates with Git for commit/branch capture and repository context, with cloud/object stores for artifact transfer, with GLaaS for global lineage publication, and with telemetry for coarse usage counters. These integrations are isolated behind dedicated adapters so core lineage logic stays environment-agnostic.

11. Auth, scope, and labels

Authentication and publication scope are first-class boundaries: auth/login manage GLaaS identity, scope determines where lineage is published, and label manages local metadata that can later be synchronized. This keeps user identity and project ownership separate from the provenance graph itself.

12. Build and packaging boundary

The project is a Python package plus Rust workspace, with GitHub Actions building and publishing wheels that include native binaries. Editable/source installs need the Rust artifacts built separately, which is why the repo includes staging scripts and binary-copy actions.

13. What to diagram

The most important diagram is a left-to-right flow from CLI command → application service → core model/repository → DB and integrations → optional publish/download/reproduce endpoints, with the Rust tracer/runtime path drawn underneath run. A second boundary should show the local system versus external services: filesystem, Git, cloud storage, telemetry, and GLaaS.