# dvorobiev/copeer architecture

> Architecture diagram of the GitHub repository dvorobiev/copeer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dvorobiev/copeer
- Repository: https://github.com/dvorobiev/copeer

## Overview

#### Purpose and entry points
Copeer is a Python CLI for migrating large media archives across multiple destination disks. It copies files in parallel and packages detected image sequences (`.dpx`, `.exr`, `.tif`) into destination-side TAR archives. Primary entry points are `copeer.py` for the full migration workflow, `copeer_lite.py` for a reduced variant, and `copeer_auditor.py` for post-run recovery and verification.

#### Main migration orchestration
`copeer.py` owns the interactive end-to-end run: parse CLI options, load or create runtime configuration, build a copy plan, display it for confirmation, execute work, and persist operational records. It supports two input boundaries: a CSV manifest or recursive source-directory scanning. `copeer.py` is the authoritative full-featured execution path.

#### Planning and source normalization
The planner converts either manifest rows or discovered filesystem objects into transferable units. Regular files become copy tasks; recognized frame sequences are grouped and represented as TAR-producing tasks. This stage establishes source-relative paths and destination layout before any writes, allowing dry-run plan generation and operator approval. Relevant implementation: `copeer.py`, `copeer_lite.py`.

#### Multi-disk destination allocation
Destination mount points, destination root, capacity threshold, and worker count are configuration-driven. Tasks are assigned across disks using an improved round-robin strategy with a priority pool, while disk usage is monitored to avoid filling targets beyond the configured threshold. This is the principal infrastructure boundary: local or mounted source storage is read, while multiple mounted destination volumes receive copies and archives. Relevant implementation: `copeer.py`.

#### Concurrent transfer and archive execution
The execution engine uses `ThreadPoolExecutor` to process independent tasks concurrently. File transfer delegates to `rsync`, including per-file live progress parsing for the terminal dashboard. Sequence tasks create TAR archives directly at the chosen destination rather than first copying individual frames. Task failures are counted and recorded rather than repeatedly printed into the UI. Relevant implementation: `copeer.py`; external runtime dependency: `rsync`.

#### Interactive observability and resumability
The full runner provides a `rich` terminal UI showing the proposed plan, aggregate progress, per-transfer `rsync` progress, worker/disk association, disk fill state, and error counts. Persistent mapping and state CSVs form the resume boundary: already completed work can be skipped after interruption. Dry runs write their mapping to a separate `dry_run_mapping.csv`, avoiding contamination of production mapping records. Relevant implementation: `copeer.py`.

#### Audit and recovery pipeline
`copeer_auditor.py` consumes artifacts from migration runs rather than performing transfers. Its `merge` command consolidates multiple mapping CSVs and removes duplicates; `analyze` compares an original input list against copier state to generate a retry manifest; `verify` checks that mapped destination paths physically exist and emits a missing-path log. The generated retry CSV feeds back into `copeer.py`, closing the recovery loop. Relevant implementation: `copeer_auditor.py`, `copeer.py`.

#### Runtime and dependencies
The repository is a Python command-line tool managed with uv-style project metadata and requirements. It relies on terminal capabilities supplied by `rich`, OS filesystem/mount semantics, Python threading, and the external `rsync` executable. Dependency declarations are in `requirements.txt`; project metadata is in `pvproject.toml`.

Main entry points: [`copeer.py`](https://github.com/dvorobiev/copeer/blob/main/copeer.py) (Concurrent executor), [`copeer_auditor.py`](https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py) (Auditor CLI), [`copeer_lite.py`](https://github.com/dvorobiev/copeer/blob/main/copeer_lite.py) (Lite migration CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["CLI entry points"]
  node_full_cli["Full migration CLI<br/>Python CLI<br/>[copeer.py]"]
  node_lite_cli["Lite migration CLI<br/>Python CLI<br/>[copeer_lite.py]"]
  node_auditor_cli["Auditor CLI<br/>Python CLI<br/>[copeer_auditor.py]"]
  node_requirements["Runtime dependencies<br/>dependency manifest<br/>[requirements.txt]"]
  node_project_metadata["Project metadata<br/>project manifest<br/>[pvproject.toml]"]
end

subgraph group_migration["Migration workflow"]
  node_runtime_config["Runtime configuration<br/>config state<br/>[copeer.py]"]
  node_input_manifest["CSV manifest<br/>input CSV<br/>[copeer.py]"]
  node_source_scan["Source directory scan<br/>filesystem discovery<br/>[copeer.py]"]
  node_planner["Copy planner<br/>planning layer<br/>[copeer.py]"]
  node_task_types["Copy and TAR tasks<br/>transfer tasks<br/>[copeer.py]"]
  node_plan_review["Plan review and dry run<br/>operator boundary<br/>[copeer.py]"]
end

subgraph group_storage["Storage and execution"]
  node_disk_allocator["Multi-disk allocator<br/>allocation layer<br/>[copeer.py]"]
  node_executor["Concurrent executor<br/>ThreadPoolExecutor<br/>[copeer.py]"]
  node_rsync{{"rsync<br/>external executable"}}
  node_destination_volumes[("Destination volumes<br/>mounted storage")]
  node_terminal_ui["Rich terminal dashboard<br/>terminal UI<br/>[copeer.py]"]
  node_run_records["Mapping and state CSVs<br/>persistent records<br/>[copeer.py]"]
end

subgraph group_recovery["Audit and recovery"]
  node_merge["Merge mappings<br/>audit command<br/>[copeer_auditor.py]"]
  node_analyze["Analyze retry work<br/>audit command<br/>[copeer_auditor.py]"]
  node_retry_manifest["Retry manifest<br/>generated CSV<br/>[copeer_auditor.py]"]
  node_verify["Verify destinations<br/>audit command<br/>[copeer_auditor.py]"]
end

node_full_cli -->|"loads or creates"| node_runtime_config
node_full_cli -->|"accepts"| node_input_manifest
node_full_cli -->|"scans"| node_source_scan
node_lite_cli -->|"uses"| node_planner
node_input_manifest -->|"normalizes"| node_planner
node_source_scan -->|"normalizes"| node_planner
node_planner -->|"builds"| node_task_types
node_task_types -->|"proposes"| node_plan_review
node_plan_review -->|"approved tasks"| node_disk_allocator
node_runtime_config -->|"capacity and mounts"| node_disk_allocator
node_disk_allocator -->|"assigned tasks"| node_executor
node_executor -->|"file copies"| node_rsync
node_executor -->|"writes TAR archives"| node_destination_volumes
node_rsync -->|"writes files"| node_destination_volumes
node_executor -->|"records outcomes"| node_run_records
node_run_records -.->|"skips completed work"| node_planner
node_executor -->|"reports progress"| node_terminal_ui
node_disk_allocator -->|"reports disk state"| node_terminal_ui
node_auditor_cli -->|"merge command"| node_merge
node_auditor_cli -->|"analyze command"| node_analyze
node_auditor_cli -->|"verify command"| node_verify
node_run_records -->|"mapping inputs"| node_merge
node_run_records -->|"copier state"| node_analyze
node_analyze -->|"generates"| node_retry_manifest
node_retry_manifest -.->|"retries through"| node_full_cli
node_run_records -->|"mapped paths"| node_verify

click node_full_cli "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_lite_cli "https://github.com/dvorobiev/copeer/blob/main/copeer_lite.py"
click node_auditor_cli "https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py"
click node_runtime_config "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_input_manifest "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_source_scan "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_planner "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_task_types "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_plan_review "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_disk_allocator "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_executor "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_terminal_ui "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_run_records "https://github.com/dvorobiev/copeer/blob/main/copeer.py"
click node_merge "https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py"
click node_analyze "https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py"
click node_retry_manifest "https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py"
click node_verify "https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py"
click node_requirements "https://github.com/dvorobiev/copeer/blob/main/requirements.txt"
click node_project_metadata "https://github.com/dvorobiev/copeer/blob/main/pvproject.toml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_full_cli,node_lite_cli,node_auditor_cli,node_requirements,node_project_metadata toneBlue
class node_runtime_config,node_input_manifest,node_source_scan,node_planner,node_task_types,node_plan_review toneAmber
class node_disk_allocator,node_executor,node_rsync,node_destination_volumes,node_terminal_ui,node_run_records toneMint
class node_merge,node_analyze,node_retry_manifest,node_verify toneRose
```

## Components

### CLI entry points

- [Full migration CLI](https://github.com/dvorobiev/copeer/blob/main/copeer.py): Python CLI · Authoritative interactive migration workflow. · `copeer.py`
- [Lite migration CLI](https://github.com/dvorobiev/copeer/blob/main/copeer_lite.py): Python CLI · Reduced planning and migration variant. · `copeer_lite.py`
- [Auditor CLI](https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py): Python CLI · Merge, analyze, and verify recovery artifacts. · `copeer_auditor.py`
- [Runtime dependencies](https://github.com/dvorobiev/copeer/blob/main/requirements.txt): dependency manifest · Declares Python dependencies including rich. · `requirements.txt`
- [Project metadata](https://github.com/dvorobiev/copeer/blob/main/pvproject.toml): project manifest · `pvproject.toml`

### Migration workflow

- [Runtime configuration](https://github.com/dvorobiev/copeer/blob/main/copeer.py): config state · Defines mounts, root, capacity threshold, and workers. · `copeer.py`
- [CSV manifest](https://github.com/dvorobiev/copeer/blob/main/copeer.py): input CSV · `copeer.py`
- [Source directory scan](https://github.com/dvorobiev/copeer/blob/main/copeer.py): filesystem discovery · `copeer.py`
- [Copy planner](https://github.com/dvorobiev/copeer/blob/main/copeer.py): planning layer · Normalizes inputs into destination-aware transferable tasks. · `copeer.py`
- [Copy and TAR tasks](https://github.com/dvorobiev/copeer/blob/main/copeer.py): transfer tasks · Files copy directly; image sequences become TAR tasks. · `copeer.py`
- [Plan review and dry run](https://github.com/dvorobiev/copeer/blob/main/copeer.py): operator boundary · Displays the plan for approval before writes. · `copeer.py`

### Storage and execution

- [Multi-disk allocator](https://github.com/dvorobiev/copeer/blob/main/copeer.py): allocation layer · Assigns tasks using capacity-aware round robin. · `copeer.py`
- [Concurrent executor](https://github.com/dvorobiev/copeer/blob/main/copeer.py): ThreadPoolExecutor · Runs independent tasks across configured workers. · `copeer.py`
- **rsync**: external executable · Copies files and supplies live transfer progress.
- **Destination volumes**: mounted storage · Multiple disks receive copied files and TAR archives.
- [Rich terminal dashboard](https://github.com/dvorobiev/copeer/blob/main/copeer.py): terminal UI · Shows plan, progress, disk state, and failures. · `copeer.py`
- [Mapping and state CSVs](https://github.com/dvorobiev/copeer/blob/main/copeer.py): persistent records · Resume boundary for completed work; dry runs use separate mapping. · `copeer.py`

### Audit and recovery

- [Merge mappings](https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py): audit command · Consolidates mapping CSVs and removes duplicates. · `copeer_auditor.py`
- [Analyze retry work](https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py): audit command · Compares original input against copier state. · `copeer_auditor.py`
- [Retry manifest](https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py): generated CSV · Feeds failed or incomplete work back to migration. · `copeer_auditor.py`
- [Verify destinations](https://github.com/dvorobiev/copeer/blob/main/copeer_auditor.py): audit command · Checks mapped destination paths and logs missing paths. · `copeer_auditor.py`

## Connections

- Full migration CLI → Runtime configuration: loads or creates
- Full migration CLI → CSV manifest: accepts
- Full migration CLI → Source directory scan: scans
- Lite migration CLI → Copy planner: uses
- CSV manifest → Copy planner: normalizes
- Source directory scan → Copy planner: normalizes
- Copy planner → Copy and TAR tasks: builds
- Copy and TAR tasks → Plan review and dry run: proposes
- Plan review and dry run → Multi-disk allocator: approved tasks
- Runtime configuration → Multi-disk allocator: capacity and mounts
- Multi-disk allocator → Concurrent executor: assigned tasks
- Concurrent executor → rsync: file copies
- Concurrent executor → Destination volumes: writes TAR archives
- rsync → Destination volumes: writes files
- Concurrent executor → Mapping and state CSVs: records outcomes
- Mapping and state CSVs → Copy planner: skips completed work
- Concurrent executor → Rich terminal dashboard: reports progress
- Multi-disk allocator → Rich terminal dashboard: reports disk state
- Auditor CLI → Merge mappings: merge command
- Auditor CLI → Analyze retry work: analyze command
- Auditor CLI → Verify destinations: verify command
- Mapping and state CSVs → Merge mappings: mapping inputs
- Mapping and state CSVs → Analyze retry work: copier state
- Analyze retry work → Retry manifest: generates
- Retry manifest → Full migration CLI: retries through
- Mapping and state CSVs → Verify destinations: mapped paths