# paritytech/substrate-archive architecture

> Architecture diagram of the GitHub repository paritytech/substrate-archive, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paritytech/substrate-archive
- Repository: https://github.com/paritytech/substrate-archive

## Overview

Based on our analysis of the file tree and README, this project is a blockchain indexing engine specifically for Substrate-based chains. Its purpose is to run alongside a Substrate node that runs with archival pruning, index blocks, state, and extrinsics, and export that data into a PostgreSQL database. There are several components in the project (for example, the CLI tools and backend processing modules) that work together to “archive” or index blockchain data.

##### Main Components
- Substrate Node (with Pruning set to ‘archive’): This is the upstream chain data provider.
- Substrate-Archive CLI: There are several CLI tools (e.g., node-template-archive and polkadot-archive in the “bin” directory) that are used to interact with the blockchain node and perform indexing.
- Archive Backend: The “substrate-archive-backend” directory contains the core libraries/modules that handle database writing, querying, and data representation. It includes components like the “frontend” interface (likely for external client queries) and “read_only_backend” components.
- Archive Engine: The “substrate-archive” directory itself holds the core engine that organizes workers and actors (see the “actors/” subdirectories) which handle concurrent tasks such as processing blocks, extrinsics, and storage changes.
- Work Queue: The “work-queue” directory appears to include utilities for asynchronous job handling and background processing.
- Database: PostgreSQL is a central element where the indexed data is stored. There are scripts and SQL files (in the “scripts” directory and the “migrations” subfolder of “substrate-archive”) that set up the database schema.
- WASM Tracing: The “wasm-tracing” directory suggests that the project also processes or validates runtime WASM code for chains like Kusama, Polkadot, and Westend.
- Auxiliary Components: There are configuration files (e.g., “archive.toml”, “Cargo.toml”, “docker-compose.yaml”, and various workflow files) that define how the system is built, tested, and deployed.

##### Relationships and Data Flow
- The Substrate Node feeds blockchain data (blocks, extrinsics, state) to the indexing CLI tools.
- The CLI and engine interact with the backend modules to parse, decode, and process the incoming data.
- Processed data is then sent to PostgreSQL – representing a clear pipeline from data ingestion to persistence.
- The Work-Queue components manage asynchronous processing, ensuring tasks like jobs for block processing or database updates run concurrently.
- The WASM Tracing module indicates that runtime WASM binaries are used or inspected, likely during processing of on-chain metadata or state.

##### Architectural Patterns and Design Principles
- Modular Structure: The project is divided into multiple directories to separate concerns (CLI tools, backend processing, work queues, WASM tracing, and testing utilities).
- Actor/Worker Model: The “actors” subdirectories imply components orchestrated as independent workers handling specific tasks (e.g., processing blocks, updating the database, aggregating storage).
- Extensibility: With examples provided (e.g., node-template-archive) and a self-contained CLI, the system is designed to be extended to support different chains (Polkadot, Kusama, Westend).
- Layered Communication: There is a clear separation between the ingestion layer (CLI and engine) and the persistence layer (database interaction modules) along with additional layers like the frontend/query layer.

##### Relevant Technologies/Frameworks
- Rust: The entire project is written in Rust (as indicated by Cargo.toml files and Rust source files).
- PostgreSQL: Used as the target SQL database.
- RocksDB (via Substrate node backed by RocksDB): Mentioned in troubleshooting notes for “too many open files.”
- Docker: Docker compose and initialization scripts are provided.
- GitHub Actions: Numerous workflows (lint, archive, test, release, etc.) indicate CI/CD integration.

##### Elements
- From Substrate Node -> CLI Tools -> Archive Engine.
- From Archive Engine (workers/actors) -> Database (PostgreSQL).

##### Final Recommendations
- Tailor the layout to highlight the pipeline from blockchain data ingestion, through processing (using rapid, asynchronous work queues and actor models), into a persistently stored PostgreSQL system.
- Though the system is split into many smaller binaries and libraries, emphasize their collaborative role in one unified archival engine.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Data Ingestion Layer
  subgraph "Data Ingestion Layer"
    SN["Substrate Node (Archive Pruning)"]:::ingestion
    subgraph "CLI Tools"
      C1["node-template-archive"]:::ingestion
      C2["polkadot-archive"]:::ingestion
    end
  end

  SN --> C1
  SN --> C2

  %% Data Processing Layer
  subgraph "Data Processing Layer"
    E["Archive Engine"]:::processing
    WB["WASM Tracing"]:::processing
    subgraph "Worker/Actor Model"
      A1["Actors/Workers"]:::processing
    end
    WQ["Work Queue"]:::processing
    B["Archive Backend"]:::processing
  end

  C1 --> E
  C2 --> E
  E --> A1
  E --> WQ
  E --> WB
  E --> B

  %% Storage Layer
  subgraph "Storage Layer"
    DB["PostgreSQL Database"]:::storage
    DS1["SQL Script: docker_postgres_init.sql"]:::storage
    DS2["Config: postgres.conf"]:::storage
    DM["Migrations (src/migrations)"]:::storage
  end

  B --> DB
  DS1 --> DB
  DS2 --> DB
  DM --> DB

  %% Auxiliary & Configuration Components
  subgraph "Auxiliary & Config"
    AC1["docker-compose.yaml"]:::config
    AC2["Cargo.toml"]:::config
    AC3["archive.toml"]:::config
  end

  %% Legend
  subgraph "Legend"
    L1["Ingestion Layer: Light Blue"]:::legend
    L2["Processing Layer: Light Green"]:::legend
    L3["Storage Layer: Light Yellow"]:::legend
    L4["Auxiliary/Config: Light Grey"]:::legend
  end

  %% Click events for component mappings
  click C1 "https://github.com/paritytech/substrate-archive/tree/master/bin/node-template-archive" "Go to node-template-archive"
  click C2 "https://github.com/paritytech/substrate-archive/tree/master/bin/polkadot-archive" "Go to polkadot-archive"
  click B "https://github.com/paritytech/substrate-archive/tree/master/substrate-archive-backend" "Go to Archive Backend directory"
  click E "https://github.com/paritytech/substrate-archive/tree/master/substrate-archive" "Go to Archive Engine directory"
  click WQ "https://github.com/paritytech/substrate-archive/tree/master/work-queue" "Go to Work Queue directory"
  click DM "https://github.com/paritytech/substrate-archive/tree/master/substrate-archive/src/migrations" "Go to Migrations directory"
  click DS1 "https://github.com/paritytech/substrate-archive/blob/master/scripts/docker_postgres_init.sql" "Go to SQL script"
  click DS2 "https://github.com/paritytech/substrate-archive/blob/master/scripts/postgres.conf" "Go to Postgres config"
  click WB "https://github.com/paritytech/substrate-archive/tree/master/wasm-tracing" "Go to WASM Tracing directory"
  click AC1 "https://github.com/paritytech/substrate-archive/blob/master/docker-compose.yaml" "Go to docker-compose.yaml"
  click AC2 "https://github.com/paritytech/substrate-archive/blob/master/Cargo.toml" "Go to Cargo.toml"
  click AC3 "https://github.com/paritytech/substrate-archive/blob/master/archive.toml" "Go to archive.toml"

  %% Class styles
  classDef ingestion fill:#add8e6,stroke:#333,stroke-width:2px;
  classDef processing fill:#90ee90,stroke:#333,stroke-width:2px;
  classDef storage fill:#ffffcc,stroke:#333,stroke-width:2px;
  classDef config fill:#d3d3d3,stroke:#333,stroke-width:2px;
  classDef legend fill:#ffffff,stroke:#000,stroke-dasharray: 5, 5, stroke-width:2px;
```
