# elixir-europe/plant-brapi-etl-faidare architecture

> Architecture diagram of the GitHub repository elixir-europe/plant-brapi-etl-faidare, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/elixir-europe/plant-brapi-etl-faidare
- Repository: https://github.com/elixir-europe/plant-brapi-etl-faidare

## Overview

This is a BrAPI ETL for plant-breeding data. Its principal workflow discovers and fetches source records, stores extracted JSON, then transforms them into Elasticsearch bulk documents or JSON-LD/RDF for Virtuoso. The graph also preserves the documented indexing and RDF loading capabilities. Configuration and source descriptions govern the run. The main dispatch and several transform relationships are visible in excerpts; some downstream RDF and loading details are documented but not shown in sampled runtime code.

Main entry points: [`main.py`](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/main.py) (ETL entry point), [`etl/transform/datadiscovery_cards.py`](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/datadiscovery_cards.py) (ES document transform), [`etl/extract/brapi.py`](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/extract/brapi.py) (Source extraction), [`etl/transform/jsonld.py`](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/jsonld.py) (JSON-LD transform)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Run configuration"]
  node_main["ETL entry point<br/>[main.py]"]
  node_cli["CLI options<br/>[cli.py]"]
  node_config["Run configuration<br/>[config.py]"]
  node_sources["Source definitions"]
end

subgraph group_extract["BrAPI extraction"]
  node_brapi["BrAPI client<br/>[brapi.py]"]
  node_extractor["Source extraction<br/>[brapi.py]"]
  node_store["JSON storage<br/>[store.py]"]
  node_rawdata["Extracted JSON"]
end

subgraph group_es["Elasticsearch documents"]
  node_cards["ES document transform"]
  node_cardfields["Card field transforms<br/>[transform_cards.py]"]
  node_discovery["Data discovery cards"]
  node_bulkdata["Bulk JSON output"]
  node_elasticsearch[("Elasticsearch")]
end

subgraph group_semantic["Linked data"]
  node_jsonld["JSON-LD transform<br/>[jsonld.py]"]
  node_rdf["RDF transform<br/>[rdf.py]"]
  node_rdfdata["RDF output"]
  node_virtuoso[("Virtuoso")]
  node_transformutils["Transform utilities<br/>[utils.py]"]
end

node_operator(("ETL operator"))

node_operator -->|"starts"| node_main
node_main -->|"parses options"| node_cli
node_main -->|"loads configuration"| node_config
node_config -->|"loads source files"| node_sources
node_main -->|"dispatches extraction"| node_extractor
node_extractor -->|"fetches endpoints"| node_brapi
node_extractor -.->|"uses storage"| node_store
node_extractor -.->|"writes JSON"| node_rawdata
node_main -->|"dispatches transform"| node_cards
node_cards -->|"reads JSON"| node_rawdata
node_cards -->|"calls transforms"| node_cardfields
node_cards -->|"generates cards"| node_discovery
node_cards -->|"writes bulk JSON"| node_bulkdata
node_bulkdata -->|"loads documents"| node_elasticsearch
node_main -->|"dispatches transform"| node_jsonld
node_jsonld -.->|"reads JSON"| node_rawdata
node_jsonld -->|"uses URI helpers"| node_transformutils
node_jsonld -.->|"produces RDF"| node_rdf
node_rdf -.->|"writes RDF"| node_rdfdata
node_rdfdata -->|"loads triples"| node_virtuoso

click node_main "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/main.py"
click node_cli "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/cli.py"
click node_config "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/config.py"
click node_sources "https://github.com/elixir-europe/plant-brapi-etl-faidare/tree/master/sources"
click node_brapi "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/common/brapi.py"
click node_extractor "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/extract/brapi.py"
click node_store "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/common/store.py"
click node_cards "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/datadiscovery_cards.py"
click node_cardfields "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/transform_cards.py"
click node_discovery "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/generate_datadiscovery.py"
click node_jsonld "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/jsonld.py"
click node_rdf "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/rdf.py"
click node_transformutils "https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_cli,node_config,node_sources toneBlue
class node_brapi,node_extractor,node_store,node_rawdata toneAmber
class node_cards,node_cardfields,node_discovery,node_bulkdata,node_elasticsearch toneMint
class node_jsonld,node_rdf,node_rdfdata,node_virtuoso,node_transformutils toneRose
class node_operator toneIndigo
```

## Components

### Run configuration

- [ETL entry point](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/main.py): `main.py`
- [CLI options](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/cli.py): `etl/cli.py`
- [Run configuration](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/config.py): `etl/config.py`
- [Source definitions](https://github.com/elixir-europe/plant-brapi-etl-faidare/tree/master/sources): `sources`

### BrAPI extraction

- [BrAPI client](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/common/brapi.py): `etl/common/brapi.py`
- [Source extraction](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/extract/brapi.py): `etl/extract/brapi.py`
- [JSON storage](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/common/store.py): `etl/common/store.py`
- **Extracted JSON**

### Elasticsearch documents

- [ES document transform](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/datadiscovery_cards.py): `etl/transform/datadiscovery_cards.py`
- [Card field transforms](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/transform_cards.py): `etl/transform/transform_cards.py`
- [Data discovery cards](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/generate_datadiscovery.py): `etl/transform/generate_datadiscovery.py`
- **Bulk JSON output**
- **Elasticsearch**

### Linked data

- [JSON-LD transform](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/jsonld.py): `etl/transform/jsonld.py`
- [RDF transform](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/rdf.py): `etl/transform/rdf.py`
- **RDF output**
- **Virtuoso**
- [Transform utilities](https://github.com/elixir-europe/plant-brapi-etl-faidare/blob/master/etl/transform/utils.py): `etl/transform/utils.py`

### Other components

- **ETL operator**

## Connections

- ETL operator → ETL entry point: starts (evidence: `README.md`)
- ETL entry point → CLI options: parses options (evidence: `main.py`)
- ETL entry point → Run configuration: loads configuration (evidence: `main.py`)
- Run configuration → Source definitions: loads source files (evidence: `etl/config.py`)
- ETL entry point → Source extraction: dispatches extraction (evidence: `main.py`)
- Source extraction → BrAPI client: fetches endpoints (evidence: `etl/extract/brapi.py`)
- Source extraction → JSON storage: uses storage (evidence: `etl/extract/brapi.py`)
- Source extraction → Extracted JSON: writes JSON
- ETL entry point → ES document transform: dispatches transform (evidence: `main.py`)
- ES document transform → Extracted JSON: reads JSON (evidence: `etl/transform/datadiscovery_cards.py`)
- ES document transform → Card field transforms: calls transforms (evidence: `etl/transform/datadiscovery_cards.py`)
- ES document transform → Data discovery cards: generates cards (evidence: `etl/transform/datadiscovery_cards.py`)
- ES document transform → Bulk JSON output: writes bulk JSON (evidence: `etl/transform/datadiscovery_cards.py`)
- Bulk JSON output → Elasticsearch: loads documents (evidence: `README.md`)
- ETL entry point → JSON-LD transform: dispatches transform (evidence: `main.py`)
- JSON-LD transform → Extracted JSON: reads JSON
- JSON-LD transform → Transform utilities: uses URI helpers (evidence: `etl/transform/jsonld.py`)
- JSON-LD transform → RDF transform: produces RDF (evidence: `README.md`)
- RDF transform → RDF output: writes RDF
- RDF output → Virtuoso: loads triples (evidence: `README.md`)
