# datosgobes/ckanext-dge-harvest architecture

> Architecture diagram of the GitHub repository datosgobes/ckanext-dge-harvest, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datosgobes/ckanext-dge-harvest
- Repository: https://github.com/datosgobes/ckanext-dge-harvest

## Overview

This CKAN extension federates remote RDF catalogs into CKAN using DCAT-AP-ES and NTI-RISP harvester/profile implementations. The main workflow is harvest dispatch, catalog retrieval and parsing, validation, then importing metadata; it also exports catalog data as RDF/CSV, maintains a Virtuoso-backed RDF catalog, and exposes structured harvest reports. The README establishes the major capabilities and integrations; sampled source confirms RDF processing, report persistence/querying, and source-clear behavior, but does not expose every orchestration call.

Main entry points: [`ckanext/dge_harvest/harvesters/dge_harvester.py`](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_harvester.py) (Harvester dispatch), [`ckanext/dge_harvest/harvesters/dge_dcat_ap_es_harvester.py`](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_dcat_ap_es_harvester.py) (DCAT-AP-ES harvester), [`ckanext/dge_harvest/harvesters/dge_nti_harvester.py`](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_nti_harvester.py) (NTI harvester), [`ckanext/dge_harvest/rdf_store`](https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/rdf_store) (RDF store access)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_harvest["Harvest workflow"]
  node_harvester["Harvester dispatch<br/>[dge_harvester.py]"]
  node_dcat_harvester["DCAT-AP-ES harvester"]
  node_nti_harvester["NTI harvester"]
  node_profiles["RDF profiles"]
  node_validation["RDF validation"]
  node_import["Metadata import"]
  node_harvest_state[("Harvest records")]
end

subgraph group_rdf["RDF catalog"]
  node_export["Catalog export"]
  node_rdf_store["RDF store access"]
end

subgraph group_reports["Harvest reporting"]
  node_report_service["Report service"]
  node_report_query["Report queries"]
  node_report_model[("Report records")]
  node_report_view["Report interface<br/>[views.py]"]
end

subgraph group_ckan["CKAN integration"]
  node_admin_cli["Harvest CLI<br/>[dge_harvester.py]"]
  node_source_tasks["Source maintenance<br/>[tasks.py]"]
end

node_operator(("CKAN operator"))
node_remote(("Remote RDF catalog"))
node_ckan_core["CKAN harvest core"]
node_virtuoso[("Virtuoso triple store")]
node_ckan_db[("CKAN database")]

node_operator -->|"starts harvest"| node_ckan_core
node_ckan_core -->|"dispatches"| node_harvester
node_harvester -->|"selects profile"| node_dcat_harvester
node_harvester -->|"selects profile"| node_nti_harvester
node_dcat_harvester -->|"fetches catalog"| node_remote
node_nti_harvester -->|"fetches catalog"| node_remote
node_dcat_harvester -->|"parses with"| node_profiles
node_nti_harvester -->|"parses with"| node_profiles
node_dcat_harvester -->|"validates data"| node_validation
node_nti_harvester -->|"validates data"| node_validation
node_dcat_harvester -->|"imports metadata"| node_import
node_nti_harvester -->|"imports metadata"| node_import
node_import -->|"writes records"| node_ckan_db
node_harvester -->|"records jobs"| node_harvest_state
node_operator -->|"runs command"| node_admin_cli
node_admin_cli -->|"creates RDF/CSV"| node_export
node_export -->|"queries catalog"| node_rdf_store
node_rdf_store -->|"queries triples"| node_virtuoso
node_rdf_store -->|"updates triples"| node_virtuoso
node_harvester -->|"records messages"| node_report_service
node_report_service -->|"persists messages"| node_report_model
node_report_query -->|"reads report rows"| node_report_model
node_report_view -->|"requests report"| node_report_query
node_operator -->|"views report"| node_report_view
node_operator -->|"clears source"| node_source_tasks
node_source_tasks -->|"updates harvest data"| node_ckan_db
node_source_tasks -->|"removes source graphs"| node_rdf_store

click node_harvester "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_harvester.py"
click node_dcat_harvester "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_dcat_ap_es_harvester.py"
click node_nti_harvester "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_nti_harvester.py"
click node_profiles "https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/profiles"
click node_validation "https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/harvesters/utils"
click node_import "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/utils/import_stage_utils.py"
click node_harvest_state "https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/model"
click node_admin_cli "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/commands/dge_harvester.py"
click node_export "https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/export"
click node_rdf_store "https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/rdf_store"
click node_report_service "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/services/report/harvest_report_service.py"
click node_report_query "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/services/report/harvest_report_query.py"
click node_report_model "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/model/dge_harvest_report.py"
click node_report_view "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/views.py"
click node_source_tasks "https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/tasks.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_harvester,node_dcat_harvester,node_nti_harvester,node_profiles,node_validation,node_import,node_harvest_state toneBlue
class node_export,node_rdf_store,node_virtuoso,node_ckan_db toneAmber
class node_report_service,node_report_query,node_report_model,node_report_view toneMint
class node_admin_cli,node_source_tasks toneRose
class node_operator,node_remote,node_ckan_core toneIndigo
```

## Components

### Harvest workflow

- [Harvester dispatch](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_harvester.py): `ckanext/dge_harvest/harvesters/dge_harvester.py`
- [DCAT-AP-ES harvester](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_dcat_ap_es_harvester.py): `ckanext/dge_harvest/harvesters/dge_dcat_ap_es_harvester.py`
- [NTI harvester](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/dge_nti_harvester.py): `ckanext/dge_harvest/harvesters/dge_nti_harvester.py`
- [RDF profiles](https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/profiles): `ckanext/dge_harvest/profiles`
- [RDF validation](https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/harvesters/utils): `ckanext/dge_harvest/harvesters/utils`
- [Metadata import](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/harvesters/utils/import_stage_utils.py): `ckanext/dge_harvest/harvesters/utils/import_stage_utils.py`
- [Harvest records](https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/model): `ckanext/dge_harvest/model`

### RDF catalog

- [Catalog export](https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/export): `ckanext/dge_harvest/export`
- [RDF store access](https://github.com/datosgobes/ckanext-dge-harvest/tree/master/ckanext/dge_harvest/rdf_store): `ckanext/dge_harvest/rdf_store`

### Harvest reporting

- [Report service](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/services/report/harvest_report_service.py): `ckanext/dge_harvest/services/report/harvest_report_service.py`
- [Report queries](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/services/report/harvest_report_query.py): `ckanext/dge_harvest/services/report/harvest_report_query.py`
- [Report records](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/model/dge_harvest_report.py): `ckanext/dge_harvest/model/dge_harvest_report.py`
- [Report interface](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/views.py): `ckanext/dge_harvest/views.py`

### CKAN integration

- [Harvest CLI](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/commands/dge_harvester.py): `ckanext/dge_harvest/commands/dge_harvester.py`
- [Source maintenance](https://github.com/datosgobes/ckanext-dge-harvest/blob/master/ckanext/dge_harvest/tasks.py): `ckanext/dge_harvest/tasks.py`

### Other components

- **CKAN operator**
- **Remote RDF catalog**
- **CKAN harvest core**
- **Virtuoso triple store**
- **CKAN database**

## Connections

- CKAN operator → CKAN harvest core: starts harvest
- CKAN harvest core → Harvester dispatch: dispatches
- Harvester dispatch → DCAT-AP-ES harvester: selects profile
- Harvester dispatch → NTI harvester: selects profile
- DCAT-AP-ES harvester → Remote RDF catalog: fetches catalog
- NTI harvester → Remote RDF catalog: fetches catalog
- DCAT-AP-ES harvester → RDF profiles: parses with
- NTI harvester → RDF profiles: parses with
- DCAT-AP-ES harvester → RDF validation: validates data
- NTI harvester → RDF validation: validates data
- DCAT-AP-ES harvester → Metadata import: imports metadata
- NTI harvester → Metadata import: imports metadata
- Metadata import → CKAN database: writes records
- Harvester dispatch → Harvest records: records jobs
- CKAN operator → Harvest CLI: runs command
- Harvest CLI → Catalog export: creates RDF/CSV
- Catalog export → RDF store access: queries catalog
- RDF store access → Virtuoso triple store: queries triples
- RDF store access → Virtuoso triple store: updates triples
- Harvester dispatch → Report service: records messages
- Report service → Report records: persists messages
- Report queries → Report records: reads report rows
- Report interface → Report queries: requests report
- CKAN operator → Report interface: views report
- CKAN operator → Source maintenance: clears source
- Source maintenance → CKAN database: updates harvest data
- Source maintenance → RDF store access: removes source graphs