How madara-alliance/madara-orchestrator works
Main Architectural Components
• Orchestrator Service: The central component managing job creation, processing, verification, and error handling. Its internal logic is organized across several modules (CLI, controllers, jobs, workers, queues, and routes).
• Job Processing Queues: Multiple queues exist for different job phases (creation, processing, verification, and failure handling). These queues are either implemented internally or leverage AWS SQS.
• External Integrations and Clients:
• Data Availability (DA) Clients (Ethereum, and other DA clients in development).
• Prover Clients (ATLANTIC, SHARP).
• Settlement Clients (e.g., Ethereum, Starknet).
• Infrastructure and Storage Services:
• MongoDB for job state, logs, and error tracking.
• AWS Services for messaging (SQS), storage (S3), alerting (SNS) and scheduling (EventBridge).
• Docker, Localstack, and other testing tools for local development.
• External Dependencies:
• Madara Node and Pathfinder, which provide blockchain data and RPC endpoints required for SNOS, state updates, etc.
Component Interactions and Data Flow
• Cron/Worker Trigger: System begins with cron jobs or scheduled triggers that enqueue work requests on the “worker trigger queue”.
• Job Creation and Processing: Workers then inspect block data (using RPC endpoints from Pathfinder or Madara) to determine what jobs to create. New jobs are enqueued into dedicated processing queues based on type (SNOS, Proving, Data Submission, State Transition).
• Verification: After processing, jobs are moved to verification queues. The system then marks jobs as complete in MongoDB once verified.
• Error Handling: Failed jobs are diverted to a separate failure handling queue and retried based on defined retry intervals.
• External Communication: Depending on the type of job, the orchestrator interacts with external services – for example, submitting proofs to prover services or sending state updates via settlement clients.
Important Architectural Patterns and Technologies
• Modular Design – The project is organized into multiple crates (orchestrator, da-clients, prover-clients, settlement-clients, utils), each responsible for clear facets of functionality.
• Queue-Based Job Processing – Emphasize the three-phase job model (creation, processing, verification) and how queues isolate different processing stages.
• Integration with AWS & External Systems – Use of AWS SQS, SNS, S3, and EventBridge alongside blockchain RPC endpoints (through Pathfinder and Madara) illustrates a hybrid on-prem/cloud design.
• Error Handling and Retry Mechanisms – Indicate that the system is built to detect failures, retry jobs, and log errors in its persistent store (MongoDB).