How elixir-europe/plant-brapi-etl-faidare works
This is a BrAPI ETL for plant-breeding data. Its principal workflow discovers and fetches source records, stores extracted JSON, then transforms them into Elasticsearch bulk documents or JSON-LD/RDF for Virtuoso. The graph also preserves the documented indexing and RDF loading capabilities. Configuration and source descriptions govern the run. The main dispatch and several transform relationships are visible in excerpts; some downstream RDF and loading details are documented but not shown in sampled runtime code.
Main entry points: main.py ETL entry point · datadiscovery_cards.py ES document transform · brapi.py Source extraction · jsonld.py JSON-LD transform
Components 19
Run configuration
- ETL entry point
main.py - CLI options
etl/cli.py - Run configuration
etl/config.py - Source definitions
sources
BrAPI extraction
- BrAPI client
etl/common/brapi.py - Source extraction
etl/extract/brapi.py - JSON storage
etl/common/store.py - Extracted JSON
Elasticsearch documents
- ES document transform
etl/transform/datadiscovery_cards.py - Card field transforms
etl/transform/transform_cards.py - Data discovery cards
etl/transform/generate_datadiscovery.py - Bulk JSON output
- Elasticsearch
Linked data
- JSON-LD transform
etl/transform/jsonld.py - RDF transform
etl/transform/rdf.py - RDF output
- Virtuoso
- Transform utilities
etl/transform/utils.py
Other components
- ETL operator
Connections 20
- ETL operator → ETL entry point: starts
README.md - ETL entry point → CLI options: parses options
main.py - ETL entry point → Run configuration: loads configuration
main.py - Run configuration → Source definitions: loads source files
etl/config.py - ETL entry point → Source extraction: dispatches extraction
main.py - Source extraction → BrAPI client: fetches endpoints
etl/extract/brapi.py - Source extraction → JSON storage: uses storage
etl/extract/brapi.py - Source extraction → Extracted JSON: writes JSON
- ETL entry point → ES document transform: dispatches transform
main.py - ES document transform → Extracted JSON: reads JSON
etl/transform/datadiscovery_cards.py - ES document transform → Card field transforms: calls transforms
etl/transform/datadiscovery_cards.py - ES document transform → Data discovery cards: generates cards
etl/transform/datadiscovery_cards.py - ES document transform → Bulk JSON output: writes bulk JSON
etl/transform/datadiscovery_cards.py - Bulk JSON output → Elasticsearch: loads documents
README.md - ETL entry point → JSON-LD transform: dispatches transform
main.py - JSON-LD transform → Extracted JSON: reads JSON
- JSON-LD transform → Transform utilities: uses URI helpers
etl/transform/jsonld.py - JSON-LD transform → RDF transform: produces RDF
README.md - RDF transform → RDF output: writes RDF
- RDF output → Virtuoso: loads triples
README.md