How agoric/agoric-subql works
Project Type and Purpose
• The indexer is designed to process blockchain events from an Agoric node (or A3P instance) and index those events into a data store. It then exposes the data (including metadata, vault statistics, and various financial metrics) through a GraphQL endpoint.
• Its usage scenarios range from serving an explorer view to powering dashboards like the inter dashboard. The README spells out how to use the indexer and troubleshoot common issues.
Analyzing the File Structure
• Top-Level Directories:
• .github: Contains GitHub Actions and workflow files (for testing and CI/CD pipelines). This highlights automated testing workflows (PR tests, liquidation testing, vault and reserve metrics testing).
• docker & docker-compose.yml: Indicate use of Docker for creating containerized environments for running the indexer and its dependencies.
• proto: Contains Protocol Buffer definitions for various Agoric-related modules (lien, swingset, vbank, vibc, vstorage) as well as definitions from cosmos. These indicate that the indexer deals with typed network messages and event formats.
• scripts: Contains Node.js scripts with MJS extensions that appear to interact with vaults, pricing, and on-chain data operations, likely aiding testing or operational workflows.
• src: Hosts the actual indexer source code. Notably, the “mappings” subfolder includes event mappings, handler files, and utilities that process blockchain events and transform them into indexed (and queryable) data.
• Other configuration files (e.g., package.json, tsconfig.json, .prettierrc, yarnrc.yml) support tooling, code style, and project building.
Main Components
a. Agoric Blockchain / Data Source:
• Blockchain node or A3P instance that provides the state-change events. Indicate that this is the external “data provider” where events originate (and note that interactions are done via RPC endpoints).
b. Indexer Ingestion & Processing Module:
• The core of the project, illustrated by the “src” directory, particularly the mappings folder.
• Sub-components for:
• Event Listener / Fetcher: A module that receives state changes from the blockchain.
• Mapping Handlers: Components in src/mappings/ that parse different event types (e.g., vaults, reserves, price feed, etc.) and apply logic to generate database entries.
• Utility Functions: In src/mappings/utils.ts, which support event processing.
c. Data Storage/Database:
• Although not explicitly detailed, indexers like these typically use a database (managed via SubQuery) to store the processed events and aggregated metrics. Represent this as a dedicated storage component.
d. GraphQL API Service:
• The indexer exposes a GraphQL endpoint (accessible at http://localhost:3000 per README). This is used for querying the indexed data.
e. External Interaction / Clients:
• Clients such as the “inter dashboard” or any subscriber to the GraphQL endpoint should be represented. Mark them as external consumers querying the API.
f. Supplementary Components:
• Docker and Containerization: Represent the docker-compose setup and docker scripts (e.g., docker/load-extensions.sh, docker/pg-Dockerfile) that support the deployment and environment orchestration.
• CI/CD Pipelines: A component symbolizing GitHub Actions workflows (from .github/workflows) that run tests such as liquidation and vaults/reserve metrics tests. This ensures code quality and smooth deployment.
Relationships and Data Flow
• From the ingestion layer, showcase processing steps (using mapping handlers) that transform the events and then store them in the database.
• How the GraphQL API queries this database and supplies data to external clients (e.g., dashboards).
• That containerization (Docker) and CI/CD (GitHub Actions) help in deploying and testing the entire system.
Architectural Patterns and Design Principles
• Event-driven design where incoming blockchain events are transformed and then stored.
• Demonstrate clear separation of concerns: data ingestion, processing, storage, and querying.
• That the use of Protocol Buffers for defining message formats (within the proto folder) is an important design decision ensuring consistency in data representation.
• Mention that the system leverages a microservices style in terms of containerization even if the components are collocated in one repository (e.g., indexer code, GraphQL API, orchestration scripts).
Technologies and Frameworks
• SubQuery: The backbone for indexing blockchain events.
• GraphQL: Used for the API that exposes the indexed data.
• Docker: For containerizing the application.
• Node.js and TypeScript: As indicated by .ts files, package.json, and tsconfig.json.
• Protocol Buffers: Shown in the proto directory for message definitions.