How fzu-av-cr/repo-tools works
Dataset Toolkit supports scientific-data acquisition and preparation, asynchronous downloads, metadata transformation, and single or bulk publication to NRP/Invenio; it also provides performance analysis and visualization. The main workflow is user-supplied data and metadata → optional preparation → download or upload orchestration → repository transfer → statistics and analysis. FRAM adds a distinct FITS-processing and metadata-enrichment branch. Relationships are limited to README statements and SOURCE INDEX references; unsampled implementations and uncertain links are not inferred.
Main entry points: bulk_download.py Bulk download · invenio/async_upload.py Shared upload engine · Delphi/async_upload.py Delphi single upload · fram_upload.py FRAM upload entrypoint
Components 25
Download
- Bulk download
download/invenio/bulk_download.py - File download
download/invenio/async_download.py - Model adapters
download/invenio/download_adapters.py - Model download entrypoints
download/invenio
Upload
- Delphi single upload
upload/Delphi/async_upload.py - Delphi bulk upload
upload/Delphi/bulk_async.py - Shared bulk engine
upload/invenio/bulk_async.py - Shared upload engine
upload/invenio/async_upload.py - Upload adapters
upload/invenio/adapters.py
FRAM processing
- FRAM upload entrypoint
upload/invenio/fram/fram_upload.py - FRAM metadata builder
upload/invenio/fram/fram_make_json.py - FITS calibration
upload/invenio/fram/calibrate.py - Image photometry
upload/invenio/fram/extract_photometry.py - Metadata correction
upload/invenio/fram/correct_jsons.py - FRAM retry upload
upload/invenio/fram/fram_upload_with_retry.py - FITS conversion runner
upload/invenio/fram/fram-fits-converter/fram_converter/cnv_runner.py - FITS converter
upload/invenio/fram/fram-fits-converter/fram_converter/cnv_converter.py - Output path mapping
upload/invenio/fram/fram-fits-converter/fram_converter/cnv_paths.py
Analysis
- Run statistics
stats - Analysis and charts
figs
Other components
- Dataset operator
- CERN Open Data
- Dataset files
data - Record metadata
metadata - NRP / Invenio
Connections 23
- Dataset operator → Bulk download: selects records
README.md - CERN Open Data → Bulk download: provides datasets
README.md - Bulk download → File download: delegates downloads
download/invenio/bulk_download.py - Bulk download → Model adapters: resolves model
download/invenio/bulk_download.py - Model adapters → Model download entrypoints: loads entrypoint
download/invenio/download_adapters.py - File download → NRP / Invenio: retrieves files
download/invenio/async_download.py - Dataset operator → Delphi bulk upload: starts bulk upload
README.md - Dataset operator → Delphi single upload: starts single upload
README.md - Delphi bulk upload → Delphi single upload: uses single uploader
upload/Delphi/bulk_async.py - Dataset operator → Shared bulk engine: starts model workflow
upload/invenio/bulk_async.py - Shared bulk engine → Shared upload engine: dispatches records
upload/invenio/bulk_async.py - Shared upload engine → Upload adapters: uses adapters
upload/invenio/async_upload.py - Delphi single upload → NRP / Invenio: publishes records
README.md - Shared upload engine → NRP / Invenio: uploads records
README.md - Dataset operator → Dataset files: prepares inputs
README.md - Dataset operator → Record metadata: prepares metadata
README.md - FRAM upload entrypoint → FITS calibration: uses calibration
upload/invenio/fram/fram_upload.py - Image photometry → FITS calibration: uses calibration
upload/invenio/fram/extract_photometry.py - FITS conversion runner → FITS converter: runs conversion
upload/invenio/fram/fram-fits-converter/fram_converter/cnv_runner.py - FITS conversion runner → Output path mapping: maps destinations
upload/invenio/fram/fram-fits-converter/fram_converter/cnv_runner.py - FRAM upload entrypoint → NRP / Invenio: publishes FRAM data
README.md - Dataset operator → Run statistics: collects run metrics
README.md - Run statistics → Analysis and charts: feeds analysis
README.md