dvorobiev/copeer

Purpose and entry points

Copeer is a Python CLI for migrating large media archives across multiple destination disks. It copies files in parallel and packages detected image sequences (.dpx, .exr, .tif) into destination-side TAR archives. Primary entry points are copeer.py for the full migration workflow, copeer_lite.py for a reduced variant, and copeer_auditor.py for post-run recovery and verification.

Main migration orchestration

copeer.py owns the interactive end-to-end run: parse CLI options, load or create runtime configuration, build a copy plan, display it for confirmation, execute work, and persist operational records. It supports two input boundaries: a CSV manifest or recursive source-directory scanning. copeer.py is the authoritative full-featured execution path.

Planning and source normalization

The planner converts either manifest rows or discovered filesystem objects into transferable units. Regular files become copy tasks; recognized frame sequences are grouped and represented as TAR-producing tasks. This stage establishes source-relative paths and destination layout before any writes, allowing dry-run plan generation and operator approval. Relevant implementation: copeer.py, copeer_lite.py.

Multi-disk destination allocation

Destination mount points, destination root, capacity threshold, and worker count are configuration-driven. Tasks are assigned across disks using an improved round-robin strategy with a priority pool, while disk usage is monitored to avoid filling targets beyond the configured threshold. This is the principal infrastructure boundary: local or mounted source storage is read, while multiple mounted destination volumes receive copies and archives. Relevant implementation: copeer.py.

Concurrent transfer and archive execution

The execution engine uses ThreadPoolExecutor to process independent tasks concurrently. File transfer delegates to rsync, including per-file live progress parsing for the terminal dashboard. Sequence tasks create TAR archives directly at the chosen destination rather than first copying individual frames. Task failures are counted and recorded rather than repeatedly printed into the UI. Relevant implementation: copeer.py; external runtime dependency: rsync.

Interactive observability and resumability

The full runner provides a rich terminal UI showing the proposed plan, aggregate progress, per-transfer rsync progress, worker/disk association, disk fill state, and error counts. Persistent mapping and state CSVs form the resume boundary: already completed work can be skipped after interruption. Dry runs write their mapping to a separate dry_run_mapping.csv, avoiding contamination of production mapping records. Relevant implementation: copeer.py.

Audit and recovery pipeline

copeer_auditor.py consumes artifacts from migration runs rather than performing transfers. Its merge command consolidates multiple mapping CSVs and removes duplicates; analyze compares an original input list against copier state to generate a retry manifest; verify checks that mapped destination paths physically exist and emits a missing-path log. The generated retry CSV feeds back into copeer.py, closing the recovery loop. Relevant implementation: copeer_auditor.py, copeer.py.

Runtime and dependencies

The repository is a Python command-line tool managed with uv-style project metadata and requirements. It relies on terminal capabilities supplied by rich, OS filesystem/mount semantics, Python threading, and the external rsync executable. Dependency declarations are in requirements.txt; project metadata is in pvproject.toml.