How paritytech/substrate-archive works
Based on our analysis of the file tree and README, this project is a blockchain indexing engine specifically for Substrate-based chains. Its purpose is to run alongside a Substrate node that runs with archival pruning, index blocks, state, and extrinsics, and export that data into a PostgreSQL database. There are several components in the project (for example, the CLI tools and backend processing modules) that work together to “archive” or index blockchain data.
Main Components
• Substrate Node (with Pruning set to ‘archive’): This is the upstream chain data provider.
• Substrate-Archive CLI: There are several CLI tools (e.g., node-template-archive and polkadot-archive in the “bin” directory) that are used to interact with the blockchain node and perform indexing.
• Archive Backend: The “substrate-archive-backend” directory contains the core libraries/modules that handle database writing, querying, and data representation. It includes components like the “frontend” interface (likely for external client queries) and “read_only_backend” components.
• Archive Engine: The “substrate-archive” directory itself holds the core engine that organizes workers and actors (see the “actors/” subdirectories) which handle concurrent tasks such as processing blocks, extrinsics, and storage changes.
• Work Queue: The “work-queue” directory appears to include utilities for asynchronous job handling and background processing.
• Database: PostgreSQL is a central element where the indexed data is stored. There are scripts and SQL files (in the “scripts” directory and the “migrations” subfolder of “substrate-archive”) that set up the database schema.
• WASM Tracing: The “wasm-tracing” directory suggests that the project also processes or validates runtime WASM code for chains like Kusama, Polkadot, and Westend.
• Auxiliary Components: There are configuration files (e.g., “archive.toml”, “Cargo.toml”, “docker-compose.yaml”, and various workflow files) that define how the system is built, tested, and deployed.
Relationships and Data Flow
• The Substrate Node feeds blockchain data (blocks, extrinsics, state) to the indexing CLI tools.
• The CLI and engine interact with the backend modules to parse, decode, and process the incoming data.
• Processed data is then sent to PostgreSQL – representing a clear pipeline from data ingestion to persistence.
• The Work-Queue components manage asynchronous processing, ensuring tasks like jobs for block processing or database updates run concurrently.
• The WASM Tracing module indicates that runtime WASM binaries are used or inspected, likely during processing of on-chain metadata or state.
Architectural Patterns and Design Principles
• Modular Structure: The project is divided into multiple directories to separate concerns (CLI tools, backend processing, work queues, WASM tracing, and testing utilities).
• Actor/Worker Model: The “actors” subdirectories imply components orchestrated as independent workers handling specific tasks (e.g., processing blocks, updating the database, aggregating storage).
• Extensibility: With examples provided (e.g., node-template-archive) and a self-contained CLI, the system is designed to be extended to support different chains (Polkadot, Kusama, Westend).
• Layered Communication: There is a clear separation between the ingestion layer (CLI and engine) and the persistence layer (database interaction modules) along with additional layers like the frontend/query layer.
Relevant Technologies/Frameworks
• Rust: The entire project is written in Rust (as indicated by Cargo.toml files and Rust source files).
• PostgreSQL: Used as the target SQL database.
• RocksDB (via Substrate node backed by RocksDB): Mentioned in troubleshooting notes for “too many open files.”
• Docker: Docker compose and initialization scripts are provided.
• GitHub Actions: Numerous workflows (lint, archive, test, release, etc.) indicate CI/CD integration.
Elements
• From Substrate Node -> CLI Tools -> Archive Engine.
• From Archive Engine (workers/actors) -> Database (PostgreSQL).
Final Recommendations
• Tailor the layout to highlight the pipeline from blockchain data ingestion, through processing (using rapid, asynchronous work queues and actor models), into a persistently stored PostgreSQL system.
• Though the system is split into many smaller binaries and libraries, emphasize their collaborative role in one unified archival engine.