# fzu-av-cr/repo-tools architecture

> Architecture diagram of the GitHub repository fzu-av-cr/repo-tools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fzu-av-cr/repo-tools
- Repository: https://github.com/fzu-av-cr/repo-tools

## Overview

Dataset Toolkit supports scientific-data acquisition and preparation, asynchronous downloads, metadata transformation, and single or bulk publication to NRP/Invenio; it also provides performance analysis and visualization. The main workflow is user-supplied data and metadata → optional preparation → download or upload orchestration → repository transfer → statistics and analysis. FRAM adds a distinct FITS-processing and metadata-enrichment branch. Relationships are limited to README statements and SOURCE INDEX references; unsampled implementations and uncertain links are not inferred.

Main entry points: [`download/invenio/bulk_download.py`](https://github.com/fzu-av-cr/repo-tools/blob/master/download/invenio/bulk_download.py) (Bulk download), [`upload/invenio/async_upload.py`](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/async_upload.py) (Shared upload engine), [`upload/Delphi/async_upload.py`](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/Delphi/async_upload.py) (Delphi single upload), [`upload/invenio/fram/fram_upload.py`](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram_upload.py) (FRAM upload entrypoint)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_download["Download"]
  node_download_bulk["Bulk download<br/>[bulk_download.py]"]
  node_download_file["File download<br/>[async_download.py]"]
  node_download_adapters["Model adapters"]
  node_model_downloads["Model download entrypoints"]
end

subgraph group_upload["Upload"]
  node_delphi_upload["Delphi single upload<br/>[async_upload.py]"]
  node_delphi_bulk["Delphi bulk upload<br/>[bulk_async.py]"]
  node_upload_bulk["Shared bulk engine<br/>[bulk_async.py]"]
  node_upload_single["Shared upload engine<br/>[async_upload.py]"]
  node_upload_adapters["Upload adapters<br/>[adapters.py]"]
end

subgraph group_fram["FRAM processing"]
  node_fram_entry["FRAM upload entrypoint<br/>[fram_upload.py]"]
  node_fram_metadata["FRAM metadata builder<br/>[fram_make_json.py]"]
  node_fram_calibration["FITS calibration<br/>[calibrate.py]"]
  node_fram_photometry["Image photometry"]
  node_fram_correction["Metadata correction<br/>[correct_jsons.py]"]
  node_fram_retry["FRAM retry upload"]
  node_converter_runner["FITS conversion runner<br/>[cnv_runner.py]"]
  node_converter["FITS converter<br/>[cnv_converter.py]"]
  node_converter_paths["Output path mapping<br/>[cnv_paths.py]"]
end

subgraph group_observability["Analysis"]
  node_stats["Run statistics"]
  node_analysis["Analysis and charts"]
end

node_user(("Dataset operator"))
node_cern["CERN Open Data"]
node_raw_data["Dataset files"]
node_source_metadata["Record metadata"]
node_invenio["NRP / Invenio"]

node_user -->|"selects records"| node_download_bulk
node_cern -->|"provides datasets"| node_download_bulk
node_download_bulk -->|"delegates downloads"| node_download_file
node_download_bulk -->|"resolves model"| node_download_adapters
node_download_adapters -->|"loads entrypoint"| node_model_downloads
node_download_file -->|"retrieves files"| node_invenio
node_user -->|"starts bulk upload"| node_delphi_bulk
node_user -->|"starts single upload"| node_delphi_upload
node_delphi_bulk -->|"uses single uploader"| node_delphi_upload
node_user -->|"starts model workflow"| node_upload_bulk
node_upload_bulk -->|"dispatches records"| node_upload_single
node_upload_single -->|"uses adapters"| node_upload_adapters
node_delphi_upload -->|"publishes records"| node_invenio
node_upload_single -->|"uploads records"| node_invenio
node_user -->|"prepares inputs"| node_raw_data
node_user -->|"prepares metadata"| node_source_metadata
node_fram_entry -->|"uses calibration"| node_fram_calibration
node_fram_photometry -->|"uses calibration"| node_fram_calibration
node_converter_runner -->|"runs conversion"| node_converter
node_converter_runner -->|"maps destinations"| node_converter_paths
node_fram_entry -->|"publishes FRAM data"| node_invenio
node_user -->|"collects run metrics"| node_stats
node_stats -->|"feeds analysis"| node_analysis

click node_raw_data "https://github.com/fzu-av-cr/repo-tools/tree/master/data"
click node_source_metadata "https://github.com/fzu-av-cr/repo-tools/tree/master/metadata"
click node_download_bulk "https://github.com/fzu-av-cr/repo-tools/blob/master/download/invenio/bulk_download.py"
click node_download_file "https://github.com/fzu-av-cr/repo-tools/blob/master/download/invenio/async_download.py"
click node_download_adapters "https://github.com/fzu-av-cr/repo-tools/blob/master/download/invenio/download_adapters.py"
click node_model_downloads "https://github.com/fzu-av-cr/repo-tools/tree/master/download/invenio"
click node_delphi_upload "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/Delphi/async_upload.py"
click node_delphi_bulk "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/Delphi/bulk_async.py"
click node_upload_bulk "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/bulk_async.py"
click node_upload_single "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/async_upload.py"
click node_upload_adapters "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/adapters.py"
click node_fram_entry "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram_upload.py"
click node_fram_metadata "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram_make_json.py"
click node_fram_calibration "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/calibrate.py"
click node_fram_photometry "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/extract_photometry.py"
click node_fram_correction "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/correct_jsons.py"
click node_fram_retry "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram_upload_with_retry.py"
click node_converter_runner "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram-fits-converter/fram_converter/cnv_runner.py"
click node_converter "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram-fits-converter/fram_converter/cnv_converter.py"
click node_converter_paths "https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram-fits-converter/fram_converter/cnv_paths.py"
click node_stats "https://github.com/fzu-av-cr/repo-tools/tree/master/stats"
click node_analysis "https://github.com/fzu-av-cr/repo-tools/tree/master/figs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_download_bulk,node_download_file,node_download_adapters,node_model_downloads toneBlue
class node_delphi_upload,node_delphi_bulk,node_upload_bulk,node_upload_single,node_upload_adapters toneAmber
class node_fram_entry,node_fram_metadata,node_fram_calibration,node_fram_photometry,node_fram_correction,node_fram_retry,node_converter_runner,node_converter,node_converter_paths toneMint
class node_stats,node_analysis toneRose
class node_user,node_cern,node_invenio toneIndigo
class node_raw_data,node_source_metadata toneTeal
```

## Components

### Download

- [Bulk download](https://github.com/fzu-av-cr/repo-tools/blob/master/download/invenio/bulk_download.py): `download/invenio/bulk_download.py`
- [File download](https://github.com/fzu-av-cr/repo-tools/blob/master/download/invenio/async_download.py): `download/invenio/async_download.py`
- [Model adapters](https://github.com/fzu-av-cr/repo-tools/blob/master/download/invenio/download_adapters.py): `download/invenio/download_adapters.py`
- [Model download entrypoints](https://github.com/fzu-av-cr/repo-tools/tree/master/download/invenio): `download/invenio`

### Upload

- [Delphi single upload](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/Delphi/async_upload.py): `upload/Delphi/async_upload.py`
- [Delphi bulk upload](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/Delphi/bulk_async.py): `upload/Delphi/bulk_async.py`
- [Shared bulk engine](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/bulk_async.py): `upload/invenio/bulk_async.py`
- [Shared upload engine](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/async_upload.py): `upload/invenio/async_upload.py`
- [Upload adapters](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/adapters.py): `upload/invenio/adapters.py`

### FRAM processing

- [FRAM upload entrypoint](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram_upload.py): `upload/invenio/fram/fram_upload.py`
- [FRAM metadata builder](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram_make_json.py): `upload/invenio/fram/fram_make_json.py`
- [FITS calibration](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/calibrate.py): `upload/invenio/fram/calibrate.py`
- [Image photometry](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/extract_photometry.py): `upload/invenio/fram/extract_photometry.py`
- [Metadata correction](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/correct_jsons.py): `upload/invenio/fram/correct_jsons.py`
- [FRAM retry upload](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram_upload_with_retry.py): `upload/invenio/fram/fram_upload_with_retry.py`
- [FITS conversion runner](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram-fits-converter/fram_converter/cnv_runner.py): `upload/invenio/fram/fram-fits-converter/fram_converter/cnv_runner.py`
- [FITS converter](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram-fits-converter/fram_converter/cnv_converter.py): `upload/invenio/fram/fram-fits-converter/fram_converter/cnv_converter.py`
- [Output path mapping](https://github.com/fzu-av-cr/repo-tools/blob/master/upload/invenio/fram/fram-fits-converter/fram_converter/cnv_paths.py): `upload/invenio/fram/fram-fits-converter/fram_converter/cnv_paths.py`

### Analysis

- [Run statistics](https://github.com/fzu-av-cr/repo-tools/tree/master/stats): `stats`
- [Analysis and charts](https://github.com/fzu-av-cr/repo-tools/tree/master/figs): `figs`

### Other components

- **Dataset operator**
- **CERN Open Data**
- [Dataset files](https://github.com/fzu-av-cr/repo-tools/tree/master/data): `data`
- [Record metadata](https://github.com/fzu-av-cr/repo-tools/tree/master/metadata): `metadata`
- **NRP / Invenio**

## Connections

- Dataset operator → Bulk download: selects records (evidence: `README.md`)
- CERN Open Data → Bulk download: provides datasets (evidence: `README.md`)
- Bulk download → File download: delegates downloads (evidence: `download/invenio/bulk_download.py`)
- Bulk download → Model adapters: resolves model (evidence: `download/invenio/bulk_download.py`)
- Model adapters → Model download entrypoints: loads entrypoint (evidence: `download/invenio/download_adapters.py`)
- File download → NRP / Invenio: retrieves files (evidence: `download/invenio/async_download.py`)
- Dataset operator → Delphi bulk upload: starts bulk upload (evidence: `README.md`)
- Dataset operator → Delphi single upload: starts single upload (evidence: `README.md`)
- Delphi bulk upload → Delphi single upload: uses single uploader (evidence: `upload/Delphi/bulk_async.py`)
- Dataset operator → Shared bulk engine: starts model workflow (evidence: `upload/invenio/bulk_async.py`)
- Shared bulk engine → Shared upload engine: dispatches records (evidence: `upload/invenio/bulk_async.py`)
- Shared upload engine → Upload adapters: uses adapters (evidence: `upload/invenio/async_upload.py`)
- Delphi single upload → NRP / Invenio: publishes records (evidence: `README.md`)
- Shared upload engine → NRP / Invenio: uploads records (evidence: `README.md`)
- Dataset operator → Dataset files: prepares inputs (evidence: `README.md`)
- Dataset operator → Record metadata: prepares metadata (evidence: `README.md`)
- FRAM upload entrypoint → FITS calibration: uses calibration (evidence: `upload/invenio/fram/fram_upload.py`)
- Image photometry → FITS calibration: uses calibration (evidence: `upload/invenio/fram/extract_photometry.py`)
- FITS conversion runner → FITS converter: runs conversion (evidence: `upload/invenio/fram/fram-fits-converter/fram_converter/cnv_runner.py`)
- FITS conversion runner → Output path mapping: maps destinations (evidence: `upload/invenio/fram/fram-fits-converter/fram_converter/cnv_runner.py`)
- FRAM upload entrypoint → NRP / Invenio: publishes FRAM data (evidence: `README.md`)
- Dataset operator → Run statistics: collects run metrics (evidence: `README.md`)
- Run statistics → Analysis and charts: feeds analysis (evidence: `README.md`)