# streamingfast/substreams architecture

> Architecture diagram of the GitHub repository streamingfast/substreams, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/streamingfast/substreams
- Repository: https://github.com/streamingfast/substreams

## Overview

##### Main Components

a) Client Interface Layer
- CLI Interface (under /cmd/substreams)
- Client Libraries (under /client)
- TUI Components (under /tui and /tui2)

b) Core Processing Layer
- Pipeline Processing (/pipeline)
- Orchestration Engine (/orchestrator)
- WASM Runtime (/wasm)
- Storage System (/storage)

c) Data Processing Components
- Block Processing
- Module Execution
- Store Management
- Index Management

##### Key Relationships to Highlight

**Client Layer → Core Processing Layer**
- ↓
**Pipeline Processor → Orchestrator**
- ↓
**WASM Runtime ← → Storage System**
- ↓
**Block Processing → Module Execution → Store Management**

##### Specific Component Details

- Request Planning
- Block Processing
- Module Execution
- Store Management
- Cache Management

b) Storage Components (use database symbols):
- Store Storage
- Index Storage
- Execution Output Storage

c) Runtime Components (use compute symbols):
- WASM Module Execution
- Module Registry
- State Management

##### Integration Points

External integration points with:
- Blockchain Data Sources
- The Graph Network
- External Consumers
- Sink Services (/sink-server)

##### Technical Details

- Protocol Buffer Definitions (/pb)
- Manifest Handling (/manifest)
- Module Execution Pipeline
- Store Operations

##### Implementation Notes

Technical annotations for:
- WASM Runtime Options (Wasmtime/Wazero)
- Storage Implementations
- Pipeline Configurations
- Module Types (Map/Store)

##### Storage and caching mechanisms

Remember to:
- Keep components grouped logically
- Clear boundaries between subsystems
- Version/date information
- Brief technical notes where relevant

This will create a comprehensive system design that accurately represents Substreams' architecture while maintaining clarity and usefulness for technical discussions.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and classes
    classDef client fill:#2374AB,color:white
    classDef core fill:#45B7D1,color:white
    classDef storage fill:#F4D03F,color:black
    classDef external fill:#808B96,color:white
    classDef runtime fill:#E74C3C,color:white
    classDef processing fill:#2ECC71,color:white

    %% Client Interface Layer
    subgraph ClientLayer
        CLI["CLI Interface"]
        ClientLib["Client Libraries"]
        TUI["TUI Components"]
    end
    
    %% Core Processing Layer
    subgraph CoreLayer
        Pipeline["Pipeline Processing"]
        Orchestrator["Orchestration Engine"]
        WasmRuntime["WASM Runtime"]
    end
    
    %% Data Processing Components
    subgraph ProcessingLayer
        BlockProc["Block Processing"]
        ModuleExec["Module Execution"]
        StoreManage["Store Management"]
        IndexManage["Index Management"]
    end
    
    %% Storage Components
    subgraph StorageLayer
        StoreStorage[("Store Storage")]
        IndexStorage[("Index Storage")]
        ExecOutput[("Execution Output")]
    end
    
    %% External Integration
    subgraph ExternalLayer
        BlockchainSrc["Blockchain Sources"]
        GraphNetwork["The Graph Network"]
        SinkServer["Sink Services"]
    end
    
    %% Technical Components
    subgraph TechnicalLayer
        ProtoBuf["Protocol Definitions"]
        Manifest["Module Management"]
        ServiceLayer["Service Layer"]
    end
    
    %% Relationships
    CLI --> Pipeline
    ClientLib --> Pipeline
    TUI --> Pipeline
    
    Pipeline --> Orchestrator
    Orchestrator --> WasmRuntime
    WasmRuntime --> BlockProc
    BlockProc --> ModuleExec
    ModuleExec --> StoreManage
    StoreManage --> IndexManage
    
    ModuleExec --> ExecOutput
    StoreManage --> StoreStorage
    IndexManage --> IndexStorage
    
    BlockchainSrc --> Pipeline
    Pipeline --> SinkServer
    ServiceLayer --> GraphNetwork
    
    Manifest --> WasmRuntime
    ProtoBuf -.-> ServiceLayer
    
    %% Click events
    click CLI "https://github.com/streamingfast/substreams/tree/develop/cmd/substreams"
    click ClientLib "https://github.com/streamingfast/substreams/tree/develop/client"
    click TUI "https://github.com/streamingfast/substreams/tree/develop/tui"
    click Pipeline "https://github.com/streamingfast/substreams/tree/develop/pipeline"
    click Orchestrator "https://github.com/streamingfast/substreams/tree/develop/orchestrator"
    click WasmRuntime "https://github.com/streamingfast/substreams/tree/develop/wasm"
    click BlockProc "https://github.com/streamingfast/substreams/tree/develop/block"
    click StoreManage "https://github.com/streamingfast/substreams/tree/develop/storage/store"
    click IndexManage "https://github.com/streamingfast/substreams/tree/develop/storage/index"
    click Manifest "https://github.com/streamingfast/substreams/tree/develop/manifest"
    click ProtoBuf "https://github.com/streamingfast/substreams/tree/develop/pb/sf/substreams"
    click ExecOutput "https://github.com/streamingfast/substreams/tree/develop/storage/execout"
    click ServiceLayer "https://github.com/streamingfast/substreams/tree/develop/service"
    click SinkServer "https://github.com/streamingfast/substreams/tree/develop/sink-server"
    
    %% Style applications
    class CLI,ClientLib,TUI client
    class Pipeline,Orchestrator,WasmRuntime core
    class StoreStorage,IndexStorage,ExecOutput storage
    class BlockchainSrc,GraphNetwork,SinkServer external
    class BlockProc,ModuleExec,StoreManage,IndexManage processing
    class ProtoBuf,Manifest,ServiceLayer core
```
