How datosgobes/ckanext-dge-harvest works
This CKAN extension federates remote RDF catalogs into CKAN using DCAT-AP-ES and NTI-RISP harvester/profile implementations. The main workflow is harvest dispatch, catalog retrieval and parsing, validation, then importing metadata; it also exports catalog data as RDF/CSV, maintains a Virtuoso-backed RDF catalog, and exposes structured harvest reports. The README establishes the major capabilities and integrations; sampled source confirms RDF processing, report persistence/querying, and source-clear behavior, but does not expose every orchestration call.
Main entry points: dge_harvester.py Harvester dispatch · dge_dcat_ap_es_harvester.py DCAT-AP-ES harvester · dge_nti_harvester.py NTI harvester · rdf_store RDF store access
Components 20
Harvest workflow
- Harvester dispatch
ckanext/dge_harvest/harvesters/dge_harvester.py - DCAT-AP-ES harvester
ckanext/dge_harvest/harvesters/dge_dcat_ap_es_harvester.py - NTI harvester
ckanext/dge_harvest/harvesters/dge_nti_harvester.py - RDF profiles
ckanext/dge_harvest/profiles - RDF validation
ckanext/dge_harvest/harvesters/utils - Metadata import
ckanext/dge_harvest/harvesters/utils/import_stage_utils.py - Harvest records
ckanext/dge_harvest/model
RDF catalog
- Catalog export
ckanext/dge_harvest/export - RDF store access
ckanext/dge_harvest/rdf_store
Harvest reporting
- Report service
ckanext/dge_harvest/services/report/harvest_report_service.py - Report queries
ckanext/dge_harvest/services/report/harvest_report_query.py - Report records
ckanext/dge_harvest/model/dge_harvest_report.py - Report interface
ckanext/dge_harvest/views.py
CKAN integration
- Harvest CLI
ckanext/dge_harvest/commands/dge_harvester.py - Source maintenance
ckanext/dge_harvest/tasks.py
Other components
- CKAN operator
- Remote RDF catalog
- CKAN harvest core
- Virtuoso triple store
- CKAN database
Connections 27
- CKAN operator → CKAN harvest core: starts harvest
- CKAN harvest core → Harvester dispatch: dispatches
- Harvester dispatch → DCAT-AP-ES harvester: selects profile
- Harvester dispatch → NTI harvester: selects profile
- DCAT-AP-ES harvester → Remote RDF catalog: fetches catalog
- NTI harvester → Remote RDF catalog: fetches catalog
- DCAT-AP-ES harvester → RDF profiles: parses with
- NTI harvester → RDF profiles: parses with
- DCAT-AP-ES harvester → RDF validation: validates data
- NTI harvester → RDF validation: validates data
- DCAT-AP-ES harvester → Metadata import: imports metadata
- NTI harvester → Metadata import: imports metadata
- Metadata import → CKAN database: writes records
- Harvester dispatch → Harvest records: records jobs
- CKAN operator → Harvest CLI: runs command
- Harvest CLI → Catalog export: creates RDF/CSV
- Catalog export → RDF store access: queries catalog
- RDF store access → Virtuoso triple store: queries triples
- RDF store access → Virtuoso triple store: updates triples
- Harvester dispatch → Report service: records messages
- Report service → Report records: persists messages
- Report queries → Report records: reads report rows
- Report interface → Report queries: requests report
- CKAN operator → Report interface: views report
- CKAN operator → Source maintenance: clears source
- Source maintenance → CKAN database: updates harvest data
- Source maintenance → RDF store access: removes source graphs