# propeller-heads/tycho-indexer architecture

> Architecture diagram of the GitHub repository propeller-heads/tycho-indexer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/propeller-heads/tycho-indexer
- Repository: https://github.com/propeller-heads/tycho-indexer

## Overview

##### Project Type and Purpose
- Tycho Indexer is a data-streaming pipeline and indexing system for blockchain (Ethereum) data, implemented as a Rust workspace with multiple crates plus a Python client.
- It provides reorg-aware, low-latency streams of protocol state deltas, backed by persistent storage and exposed via RPC.

##### File Structure
- Top-level crates:
- tycho-ethereum: blockchain-specific extractors and analyzers
- tycho-indexer: core pipeline, extractors, Protobuf definitions, gRPC/RPC service
- tycho-storage: PostgreSQL database layer (Diesel), migrations
- tycho-common: shared DTOs, traits, models
- tycho-client (Rust CLI): connects to the RPC service, buffers deltas
- tycho-client-py: Python wrapper around the Rust client and RPC
- Auxiliary directories:
- proto: Protobuf/Buf definitions for all services
- db/docker-compose.yaml & postgres.Dockerfile: local Postgres setup
- scripts: ad-hoc data-dump and comparison scripts
- CI/CD files: GitHub workflows, Dockerfile, release.config.js

##### Readme Insights
- Extractors process on-chain blocks/deltas → indexer publishes to storage and RPC.
- Clients (Rust/Python) consume RPC and streamed deltas to reconstruct protocol state.

a. On-Chain Data Source
- Ethereum network (external)
b. Extractor Layer
- tycho-ethereum extractors (account_extractor, token_analyzer)
- substreams-based ingest (proto/sf/substreams)
c. Core Processing & RPC
- tycho-indexer pipeline (extractor runners, post-processors, reorg buffer)
- gRPC/service layer (tonic)
d. Persistent Storage
- tycho-storage (PostgreSQL via Diesel)
- Migrations folder
e. Client Interfaces
- tycho-client (Rust CLI)
- tycho-client-py (Python package)

##### Relationships and Data Flows
- From extractors → tycho-indexer core (via Substreams/gRPC)
- From indexer → PostgreSQL (write)
- From indexer → RPC endpoint (publish delta streams)
- From RPC endpoint → tycho-client & tycho-client-py (read)
- Auxiliary scripts reading from storage or RPC for ad-hoc tasks.

##### Architectural Patterns
- Microservices-style crates, each with a clear API boundary
- Streaming data pipeline (Substreams, gRPC streams)
- Shared schemas via Protobuf (proto/, buf.yaml)
- Database migrations for versioned storage (tycho-storage/migrations)
- Language interoperability (Rust core, Python client)

##### Important Technologies & Frameworks
- Rust workspace with Cargo (multiple Cargo.toml files)
- Protobuf + buf (buf.yaml, proto/) for schema & codegen
- tonic (gRPC server/client)
- PostgreSQL + Diesel ORM (tycho-storage)
- Docker & docker-compose for local Postgres
- Python packaging (pyproject.toml, build_wheel.sh)

##### Final Checklist
- [ ] All modules/crates represented
- [ ] Clear labels and roles
- [ ] Protobuf/gRPC boundary shown

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External On-Chain Source
    Ethereum["Ethereum Network"]:::external

    %% Protobuf Definitions
    Proto["Protobuf Definitions (proto/)"]:::proto

    %% Extractor Layer
    subgraph "Extractor Layer"
        style Extractors fill:#cfc,stroke:#393
        Extractors[/"tycho-ethereum (Extractor Layer)"/]:::extractor
        Account["Account Extractor Module"]:::extractor
        Token["Token Analyzer Module"]:::extractor
    end

    %% Core Processing & RPC
    subgraph "Core Pipeline & RPC Service"
        style Core fill:#fcf,stroke:#939
        Indexer[/"tycho-indexer: core pipeline & RPC"/]:::core
        subgraph "Extractor & Reorg"
            ER["Extractor Runners & Reorg Buffer"]:::core
        end
        subgraph "gRPC & Services"
            GRPC["gRPC & Service Layer"]:::core
        end
        subgraph "Substreams Integration"
            SS["Substreams Integration"]:::core
        end
        subgraph "Protobuf Codegen"
            PB["Protobuf-generated Code"]:::core
        end
    end

    %% Storage Layer
    subgraph "Persistent Storage Layer"
        style Storage fill:#cff,stroke:#393
        Storage[/"tycho-storage"/]:::storage
        Migrations["Database Migrations"]:::storage
        Diesel["Diesel PostgreSQL ORM"]:::storage
    end

    %% Client Interfaces
    subgraph "Clients"
        style Clients fill:#ffc,stroke:#993
        RustClient("tycho-client (Rust CLI)"):::client
        PyClient("tycho-client-py (Python Client)"):::client
    end

    %% Auxiliary
    Scripts("scripts/ (Aux Data Scripts)"):::aux
    DBSetup["db/docker-compose.yaml"]:::aux
    PostgresFile["postgres.Dockerfile"]:::aux

    %% Data Flows
    Ethereum -->|blocks/deltas| Account
    Ethereum -->|blocks/deltas| Token
    Account -->|stream| SS
    Token -->|stream| SS
    Proto -->|schemas| Account
    Proto -->|schemas| Token
    Proto -->|schemas| PB
    Proto -->|schemas| RustClient
    Proto -->|schemas| PyClient
    SS -->|feeds| ER
    ER -->|processed deltas| Storage
    ER -->|processed deltas| GRPC
    PB -->|protobuf API| GRPC
    GRPC -->|RPC streams| RustClient
    GRPC -->|RPC streams| PyClient
    Scripts -->|ad-hoc reads| Storage
    Scripts -->|ad-hoc reads| GRPC
    DBSetup --> Storage
    PostgresFile --> Storage

    %% Click Events
    click Extractors "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-ethereum/"
    click Account "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-ethereum/src/account_extractor/"
    click Token "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-ethereum/src/token_analyzer/"
    click Proto "https://github.com/propeller-heads/tycho-indexer/tree/main/proto/"
    click Indexer "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-indexer/"
    click ER "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-indexer/src/extractor/"
    click GRPC "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-indexer/src/services/"
    click PB "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-indexer/src/pb/"
    click SS "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-indexer/src/substreams/"
    click Storage "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-storage/"
    click Migrations "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-storage/migrations/"
    click Diesel "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-storage/src/postgres/"
    click RustClient "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-client/"
    click PyClient "https://github.com/propeller-heads/tycho-indexer/tree/main/tycho-client-py/"
    click DBSetup "https://github.com/propeller-heads/tycho-indexer/blob/main/db/docker-compose.yaml"
    click PostgresFile "https://github.com/propeller-heads/tycho-indexer/blob/main/postgres.Dockerfile"
    click Scripts "https://github.com/propeller-heads/tycho-indexer/tree/main/scripts/"

    %% Styling
    classDef external fill:#ccf,stroke:#339,stroke-width:2px
    classDef extractor fill:#cfc,stroke:#393,stroke-width:2px
    classDef core fill:#fcf,stroke:#939,stroke-width:2px
    classDef storage fill:#cff,stroke:#393,stroke-width:2px
    classDef client fill:#ffc,stroke:#993,stroke-width:2px
    classDef proto fill:#ffedcc,stroke:#cc9,stroke-width:2px
    classDef aux fill:#eee,stroke:#999,stroke-width:1px
```
