How propeller-heads/tycho-indexer works
Project Type and Purpose
• Tycho Indexer is a data-streaming pipeline and indexing system for blockchain (Ethereum) data, implemented as a Rust workspace with multiple crates plus a Python client.
• It provides reorg-aware, low-latency streams of protocol state deltas, backed by persistent storage and exposed via RPC.
File Structure
• Top-level crates:
• tycho-ethereum: blockchain-specific extractors and analyzers
• tycho-indexer: core pipeline, extractors, Protobuf definitions, gRPC/RPC service
• tycho-storage: PostgreSQL database layer (Diesel), migrations
• tycho-common: shared DTOs, traits, models
• tycho-client (Rust CLI): connects to the RPC service, buffers deltas
• tycho-client-py: Python wrapper around the Rust client and RPC
• Auxiliary directories:
• proto: Protobuf/Buf definitions for all services
• db/docker-compose.yaml & postgres.Dockerfile: local Postgres setup
• scripts: ad-hoc data-dump and comparison scripts
• CI/CD files: GitHub workflows, Dockerfile, release.config.js
Readme Insights
• Extractors process on-chain blocks/deltas → indexer publishes to storage and RPC.
• Clients (Rust/Python) consume RPC and streamed deltas to reconstruct protocol state.
a. On-Chain Data Source
• Ethereum network (external)
b. Extractor Layer
• tycho-ethereum extractors (account_extractor, token_analyzer)
• substreams-based ingest (proto/sf/substreams)
c. Core Processing & RPC
• tycho-indexer pipeline (extractor runners, post-processors, reorg buffer)
• gRPC/service layer (tonic)
d. Persistent Storage
• tycho-storage (PostgreSQL via Diesel)
• Migrations folder
e. Client Interfaces
• tycho-client (Rust CLI)
• tycho-client-py (Python package)
Relationships and Data Flows
• From extractors → tycho-indexer core (via Substreams/gRPC)
• From indexer → PostgreSQL (write)
• From indexer → RPC endpoint (publish delta streams)
• From RPC endpoint → tycho-client & tycho-client-py (read)
• Auxiliary scripts reading from storage or RPC for ad-hoc tasks.
Architectural Patterns
• Microservices-style crates, each with a clear API boundary
• Streaming data pipeline (Substreams, gRPC streams)
• Shared schemas via Protobuf (proto/, buf.yaml)
• Database migrations for versioned storage (tycho-storage/migrations)
• Language interoperability (Rust core, Python client)
Important Technologies & Frameworks
• Rust workspace with Cargo (multiple Cargo.toml files)
• Protobuf + buf (buf.yaml, proto/) for schema & codegen
• tonic (gRPC server/client)
• PostgreSQL + Diesel ORM (tycho-storage)
• Docker & docker-compose for local Postgres
• Python packaging (pyproject.toml, build_wheel.sh)
Final Checklist
• [ ] All modules/crates represented
• [ ] Clear labels and roles
• [ ] Protobuf/gRPC boundary shown