How diadata-org/decentral-feeder works
Main Components
• Data Scraper – As seen in the “cmd/scrapers” directory and the pkg/scrapers package, this component is responsible for retrieving trade data from various exchanges (Binance, CoinBase, Crypto.com, GateIO, Kraken, KuCoin, Uniswap, etc.).
• Data Simulator – Located under “cmd/simulators” and “pkg/simulations”, this part seems to support simulation functions that likely complement or test the data feeder.
• Collector & Processor – The README explains that after scraping, the data is fed through a two-step process involving data collection (aggregation) and processing (applying filters like median and atomic aggregation). Look at packages such as pkg/processor, pkg/metafilters, and pkg/simulations/simulationProcessor.
• Smart Contract Interface – There is an integration with on-chain components (e.g., pkg/onchain, pkg/contracts) which are used for deploying contracts and interacting with blockchain elements.
• Configuration & External Integration – The “config” directory holds JSON files for exchange pairs, pools, and symbol identification, indicating how the system is set up for different data sources.
• Container Orchestration – The existence of Dockerfiles in the “build” folder, a “docker-compose” directory, and a dedicated README section for Docker deployment indicates a strong container-based deployment model.
• CI/CD Pipelines – The .github/workflows directory suggests automated build and deployment processes which, while not directly part of the runtime architecture, are important in the development lifecycle.
Relationships & Data Flows
• External Data Sources: Represent the various exchanges (CEXs and DEXs) from which the scraper fetches real-time trade data.
• Data Collection: Following scraping, show that the data flows into a collector module that aggregates data from different scrapers (or even simulation inputs). This step might be represented as a separate “Collector” component.
• Supporting Modules: Include a segment for “Configuration” which supplies parameters (via JSON files) to the scraper and processor, and “Monitoring” to represent the metrics push via the Pushgateway (using the required environment variables).
Architectural Patterns & Design Principles
• Micro-Component Separation: Even though the project may be run as a single container/application, its structure shows clear separation of concerns (scraping, collection, processing, on-chain interaction).
• Modularity: The division into “pkg/”, “cmd/”, and “config/” indicates modular design and ease of extensibility for additional exchanges or filters.
• Containerization: Emphasize that the project is built for a containerized environment, simplifying deployment across various machines.
• CI/CD Integration: Although not part of runtime, the inclusion of GitHub workflows shows automated testing and deployment pipelines.