# stellar/stellar-etl architecture

> Architecture diagram of the GitHub repository stellar/stellar-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stellar/stellar-etl
- Repository: https://github.com/stellar/stellar-etl

## Overview

Stellar ETL is a command-line pipeline for exporting Stellar ledger history over selected ledger ranges. Users invoke an export command, which obtains ledger data, transforms it into domain-specific records, and writes output files; batch exports can also upload to cloud storage and optionally produce Parquet. The graph preserves major export domains and the time-to-ledger utility. Source excerpts show the shared batch flow and selected transformations; other command-to-module links rely on SOURCE INDEX imports, so unsampled internal details are not inferred.

Main entry points: [`cmd/root.go`](https://github.com/stellar/stellar-etl/blob/master/cmd/root.go) (Command registry), [`internal/input`](https://github.com/stellar/stellar-etl/tree/master/internal/input) (Ledger readers), [`cmd/ledger_batch_export.go`](https://github.com/stellar/stellar-etl/blob/master/cmd/ledger_batch_export.go) (Batch export runner), [`cmd/export_assets.go`](https://github.com/stellar/stellar-etl/blob/master/cmd/export_assets.go) (Asset export)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["CLI and exports"]
  node_entry["Program entry<br/>[main.go]"]
  node_commands["Command registry<br/>[root.go]"]
  node_batch["Batch export runner"]
  node_export_ledgers["Ledger export<br/>[export_ledgers.go]"]
  node_export_transactions["Transaction export"]
  node_export_operations["Operation export"]
  node_export_effects["Effects export<br/>[export_effects.go]"]
  node_export_assets["Asset export<br/>[export_assets.go]"]
  node_export_trades["Trade export<br/>[export_trades.go]"]
  node_export_changes["Entry change export"]
  node_range_command["Time range utility"]
end

subgraph group_input["Ledger input"]
  node_ledger_input["Ledger readers"]
  node_orderbooks["Orderbook parser<br/>[orderbooks.go]"]
  node_range_finder["Time range finder<br/>[ledger_range.go]"]
end

subgraph group_transform["Domain transforms"]
  node_ledger_transform["Ledger transform<br/>[ledger.go]"]
  node_transaction_transform["Transaction transform<br/>[transaction.go]"]
  node_operation_transform["Operation transform<br/>[operation.go]"]
  node_effects_transform["Effects transform<br/>[effects.go]"]
  node_trade_transform["Trade transform<br/>[trade.go]"]
  node_contract_data_transform["Contract data transform<br/>[contract_data.go]"]
  node_contract_events_transform["Contract events transform<br/>[contract_events.go]"]
end

subgraph group_output["Output utilities"]
  node_output_utils["Export and storage utilities<br/>[main.go]"]
end

node_user(("CLI user"))
node_datastore[("Ledger datastore")]
node_gcs[("Google Cloud Storage")]

node_user -->|"runs"| node_entry
node_entry -->|"executes"| node_commands
node_user -.->|"selects command"| node_commands
node_commands -->|"dispatches batch"| node_batch
node_commands -.->|"dispatches"| node_export_ledgers
node_commands -.->|"dispatches"| node_export_transactions
node_commands -.->|"dispatches"| node_export_operations
node_commands -.->|"dispatches"| node_export_effects
node_commands -.->|"dispatches"| node_export_assets
node_commands -.->|"dispatches"| node_export_trades
node_commands -.->|"dispatches"| node_export_changes
node_commands -->|"dispatches"| node_range_command
node_export_assets -->|"starts runner"| node_batch
node_batch -->|"streams batches"| node_ledger_input
node_batch -->|"prepares backend"| node_output_utils
node_batch -->|"uploads output"| node_output_utils
node_batch -.->|"may upload"| node_gcs
node_range_command -->|"calculates range"| node_range_finder
node_range_finder -->|"reads ledger files"| node_datastore
node_export_ledgers -->|"uses input"| node_ledger_input
node_export_transactions -->|"uses input"| node_ledger_input
node_export_transactions -->|"transforms records"| node_transaction_transform
node_export_operations -->|"uses input"| node_ledger_input
node_export_operations -->|"transforms records"| node_operation_transform
node_export_effects -->|"uses input"| node_ledger_input
node_export_effects -->|"transforms records"| node_effects_transform
node_export_assets -->|"extracts operations"| node_ledger_input
node_export_assets -->|"writes entries"| node_output_utils
node_export_trades -->|"uses input"| node_ledger_input
node_export_trades -->|"transforms records"| node_trade_transform
node_export_changes -->|"uses input"| node_ledger_input
node_export_changes -->|"transforms entries"| node_operation_transform
node_orderbooks -->|"reads changes"| node_ledger_input
node_ledger_input -->|"uses transform"| node_ledger_transform
node_ledger_input -->|"uses utilities"| node_output_utils

click node_entry "https://github.com/stellar/stellar-etl/blob/master/main.go"
click node_commands "https://github.com/stellar/stellar-etl/blob/master/cmd/root.go"
click node_batch "https://github.com/stellar/stellar-etl/blob/master/cmd/ledger_batch_export.go"
click node_export_ledgers "https://github.com/stellar/stellar-etl/blob/master/cmd/export_ledgers.go"
click node_export_transactions "https://github.com/stellar/stellar-etl/blob/master/cmd/export_transactions.go"
click node_export_operations "https://github.com/stellar/stellar-etl/blob/master/cmd/export_operations.go"
click node_export_effects "https://github.com/stellar/stellar-etl/blob/master/cmd/export_effects.go"
click node_export_assets "https://github.com/stellar/stellar-etl/blob/master/cmd/export_assets.go"
click node_export_trades "https://github.com/stellar/stellar-etl/blob/master/cmd/export_trades.go"
click node_export_changes "https://github.com/stellar/stellar-etl/blob/master/cmd/export_ledger_entry_changes.go"
click node_range_command "https://github.com/stellar/stellar-etl/blob/master/cmd/get_ledger_range_from_times.go"
click node_ledger_input "https://github.com/stellar/stellar-etl/tree/master/internal/input"
click node_orderbooks "https://github.com/stellar/stellar-etl/blob/master/internal/input/orderbooks.go"
click node_range_finder "https://github.com/stellar/stellar-etl/blob/master/internal/input/ledger_range.go"
click node_ledger_transform "https://github.com/stellar/stellar-etl/blob/master/internal/transform/ledger.go"
click node_transaction_transform "https://github.com/stellar/stellar-etl/blob/master/internal/transform/transaction.go"
click node_operation_transform "https://github.com/stellar/stellar-etl/blob/master/internal/transform/operation.go"
click node_effects_transform "https://github.com/stellar/stellar-etl/blob/master/internal/transform/effects.go"
click node_trade_transform "https://github.com/stellar/stellar-etl/blob/master/internal/transform/trade.go"
click node_contract_data_transform "https://github.com/stellar/stellar-etl/blob/master/internal/transform/contract_data.go"
click node_contract_events_transform "https://github.com/stellar/stellar-etl/blob/master/internal/transform/contract_events.go"
click node_output_utils "https://github.com/stellar/stellar-etl/blob/master/internal/utils/main.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry,node_commands,node_batch,node_export_ledgers,node_export_transactions,node_export_operations,node_export_effects,node_export_assets,node_export_trades,node_export_changes,node_range_command,node_user toneBlue
class node_ledger_input,node_orderbooks,node_range_finder,node_datastore,node_gcs toneAmber
class node_ledger_transform,node_transaction_transform,node_operation_transform,node_effects_transform,node_trade_transform,node_contract_data_transform,node_contract_events_transform toneMint
class node_output_utils toneRose
```

## Components

### CLI and exports

- [Program entry](https://github.com/stellar/stellar-etl/blob/master/main.go): `main.go`
- [Command registry](https://github.com/stellar/stellar-etl/blob/master/cmd/root.go): `cmd/root.go`
- [Batch export runner](https://github.com/stellar/stellar-etl/blob/master/cmd/ledger_batch_export.go): `cmd/ledger_batch_export.go`
- [Ledger export](https://github.com/stellar/stellar-etl/blob/master/cmd/export_ledgers.go): `cmd/export_ledgers.go`
- [Transaction export](https://github.com/stellar/stellar-etl/blob/master/cmd/export_transactions.go): `cmd/export_transactions.go`
- [Operation export](https://github.com/stellar/stellar-etl/blob/master/cmd/export_operations.go): `cmd/export_operations.go`
- [Effects export](https://github.com/stellar/stellar-etl/blob/master/cmd/export_effects.go): `cmd/export_effects.go`
- [Asset export](https://github.com/stellar/stellar-etl/blob/master/cmd/export_assets.go): `cmd/export_assets.go`
- [Trade export](https://github.com/stellar/stellar-etl/blob/master/cmd/export_trades.go): `cmd/export_trades.go`
- [Entry change export](https://github.com/stellar/stellar-etl/blob/master/cmd/export_ledger_entry_changes.go): `cmd/export_ledger_entry_changes.go`
- [Time range utility](https://github.com/stellar/stellar-etl/blob/master/cmd/get_ledger_range_from_times.go): `cmd/get_ledger_range_from_times.go`

### Ledger input

- [Ledger readers](https://github.com/stellar/stellar-etl/tree/master/internal/input): `internal/input`
- [Orderbook parser](https://github.com/stellar/stellar-etl/blob/master/internal/input/orderbooks.go): `internal/input/orderbooks.go`
- [Time range finder](https://github.com/stellar/stellar-etl/blob/master/internal/input/ledger_range.go): `internal/input/ledger_range.go`

### Domain transforms

- [Ledger transform](https://github.com/stellar/stellar-etl/blob/master/internal/transform/ledger.go): `internal/transform/ledger.go`
- [Transaction transform](https://github.com/stellar/stellar-etl/blob/master/internal/transform/transaction.go): `internal/transform/transaction.go`
- [Operation transform](https://github.com/stellar/stellar-etl/blob/master/internal/transform/operation.go): `internal/transform/operation.go`
- [Effects transform](https://github.com/stellar/stellar-etl/blob/master/internal/transform/effects.go): `internal/transform/effects.go`
- [Trade transform](https://github.com/stellar/stellar-etl/blob/master/internal/transform/trade.go): `internal/transform/trade.go`
- [Contract data transform](https://github.com/stellar/stellar-etl/blob/master/internal/transform/contract_data.go): `internal/transform/contract_data.go`
- [Contract events transform](https://github.com/stellar/stellar-etl/blob/master/internal/transform/contract_events.go): `internal/transform/contract_events.go`

### Output utilities

- [Export and storage utilities](https://github.com/stellar/stellar-etl/blob/master/internal/utils/main.go): `internal/utils/main.go`

### Other components

- **CLI user**
- **Ledger datastore**
- **Google Cloud Storage**

## Connections

- CLI user → Program entry: runs (evidence: `main.go`)
- Program entry → Command registry: executes (evidence: `main.go`)
- CLI user → Command registry: selects command
- Command registry → Batch export runner: dispatches batch
- Command registry → Ledger export: dispatches
- Command registry → Transaction export: dispatches
- Command registry → Operation export: dispatches
- Command registry → Effects export: dispatches
- Command registry → Asset export: dispatches
- Command registry → Trade export: dispatches
- Command registry → Entry change export: dispatches
- Command registry → Time range utility: dispatches (evidence: `cmd/get_ledger_range_from_times.go`)
- Asset export → Batch export runner: starts runner (evidence: `cmd/export_assets.go`)
- Batch export runner → Ledger readers: streams batches (evidence: `cmd/ledger_batch_export.go`)
- Batch export runner → Export and storage utilities: prepares backend (evidence: `cmd/ledger_batch_export.go`)
- Batch export runner → Export and storage utilities: uploads output (evidence: `cmd/ledger_batch_export.go`)
- Batch export runner → Google Cloud Storage: may upload (evidence: `README.md`)
- Time range utility → Time range finder: calculates range (evidence: `cmd/get_ledger_range_from_times.go`)
- Time range finder → Ledger datastore: reads ledger files (evidence: `internal/input/ledger_range.go`)
- Ledger export → Ledger readers: uses input
- Transaction export → Ledger readers: uses input (evidence: `cmd/export_transactions.go`)
- Transaction export → Transaction transform: transforms records (evidence: `cmd/export_transactions.go`)
- Operation export → Ledger readers: uses input (evidence: `cmd/export_operations.go`)
- Operation export → Operation transform: transforms records (evidence: `cmd/export_operations.go`)
- Effects export → Ledger readers: uses input (evidence: `cmd/export_effects.go`)
- Effects export → Effects transform: transforms records (evidence: `cmd/export_effects.go`)
- Asset export → Ledger readers: extracts operations (evidence: `cmd/export_assets.go`)
- Asset export → Export and storage utilities: writes entries (evidence: `cmd/export_assets.go`)
- Trade export → Ledger readers: uses input (evidence: `cmd/export_trades.go`)
- Trade export → Trade transform: transforms records (evidence: `cmd/export_trades.go`)
- Entry change export → Ledger readers: uses input (evidence: `cmd/export_ledger_entry_changes.go`)
- Entry change export → Operation transform: transforms entries (evidence: `cmd/export_ledger_entry_changes.go`)
- Orderbook parser → Ledger readers: reads changes (evidence: `internal/input/orderbooks.go`)
- Ledger readers → Ledger transform: uses transform (evidence: `internal/input/ledger_batches.go`)
- Ledger readers → Export and storage utilities: uses utilities (evidence: `internal/input/ledger_range.go`)