How stellar/stellar-etl works
Stellar ETL is a command-line pipeline for exporting Stellar ledger history over selected ledger ranges. Users invoke an export command, which obtains ledger data, transforms it into domain-specific records, and writes output files; batch exports can also upload to cloud storage and optionally produce Parquet. The graph preserves major export domains and the time-to-ledger utility. Source excerpts show the shared batch flow and selected transformations; other command-to-module links rely on SOURCE INDEX imports, so unsampled internal details are not inferred.
Main entry points: root.go Command registry · input Ledger readers · ledger_batch_export.go Batch export runner · export_assets.go Asset export
Components 25
CLI and exports
- Program entry
main.go - Command registry
cmd/root.go - Batch export runner
cmd/ledger_batch_export.go - Ledger export
cmd/export_ledgers.go - Transaction export
cmd/export_transactions.go - Operation export
cmd/export_operations.go - Effects export
cmd/export_effects.go - Asset export
cmd/export_assets.go - Trade export
cmd/export_trades.go - Entry change export
cmd/export_ledger_entry_changes.go - Time range utility
cmd/get_ledger_range_from_times.go
Ledger input
- Ledger readers
internal/input - Orderbook parser
internal/input/orderbooks.go - Time range finder
internal/input/ledger_range.go
Domain transforms
- Ledger transform
internal/transform/ledger.go - Transaction transform
internal/transform/transaction.go - Operation transform
internal/transform/operation.go - Effects transform
internal/transform/effects.go - Trade transform
internal/transform/trade.go - Contract data transform
internal/transform/contract_data.go - Contract events transform
internal/transform/contract_events.go
Output utilities
- Export and storage utilities
internal/utils/main.go
Other components
- CLI user
- Ledger datastore
- Google Cloud Storage
Connections 35
- CLI user → Program entry: runs
main.go - Program entry → Command registry: executes
main.go - CLI user → Command registry: selects command
- Command registry → Batch export runner: dispatches batch
- Command registry → Ledger export: dispatches
- Command registry → Transaction export: dispatches
- Command registry → Operation export: dispatches
- Command registry → Effects export: dispatches
- Command registry → Asset export: dispatches
- Command registry → Trade export: dispatches
- Command registry → Entry change export: dispatches
- Command registry → Time range utility: dispatches
cmd/get_ledger_range_from_times.go - Asset export → Batch export runner: starts runner
cmd/export_assets.go - Batch export runner → Ledger readers: streams batches
cmd/ledger_batch_export.go - Batch export runner → Export and storage utilities: prepares backend
cmd/ledger_batch_export.go - Batch export runner → Export and storage utilities: uploads output
cmd/ledger_batch_export.go - Batch export runner → Google Cloud Storage: may upload
README.md - Time range utility → Time range finder: calculates range
cmd/get_ledger_range_from_times.go - Time range finder → Ledger datastore: reads ledger files
internal/input/ledger_range.go - Ledger export → Ledger readers: uses input
- Transaction export → Ledger readers: uses input
cmd/export_transactions.go - Transaction export → Transaction transform: transforms records
cmd/export_transactions.go - Operation export → Ledger readers: uses input
cmd/export_operations.go - Operation export → Operation transform: transforms records
cmd/export_operations.go - Effects export → Ledger readers: uses input
cmd/export_effects.go - Effects export → Effects transform: transforms records
cmd/export_effects.go - Asset export → Ledger readers: extracts operations
cmd/export_assets.go - Asset export → Export and storage utilities: writes entries
cmd/export_assets.go - Trade export → Ledger readers: uses input
cmd/export_trades.go - Trade export → Trade transform: transforms records
cmd/export_trades.go - Entry change export → Ledger readers: uses input
cmd/export_ledger_entry_changes.go - Entry change export → Operation transform: transforms entries
cmd/export_ledger_entry_changes.go - Orderbook parser → Ledger readers: reads changes
internal/input/orderbooks.go - Ledger readers → Ledger transform: uses transform
internal/input/ledger_batches.go - Ledger readers → Export and storage utilities: uses utilities
internal/input/ledger_range.go