# diadata-org/decentral-feeder architecture

> Architecture diagram of the GitHub repository diadata-org/decentral-feeder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/diadata-org/decentral-feeder
- Repository: https://github.com/diadata-org/decentral-feeder

## Overview

##### Main Components
- Data Scraper – As seen in the “cmd/scrapers” directory and the pkg/scrapers package, this component is responsible for retrieving trade data from various exchanges (Binance, CoinBase, Crypto.com, GateIO, Kraken, KuCoin, Uniswap, etc.).
- Data Simulator – Located under “cmd/simulators” and “pkg/simulations”, this part seems to support simulation functions that likely complement or test the data feeder.
- Collector & Processor – The README explains that after scraping, the data is fed through a two-step process involving data collection (aggregation) and processing (applying filters like median and atomic aggregation). Look at packages such as pkg/processor, pkg/metafilters, and pkg/simulations/simulationProcessor.
- Smart Contract Interface – There is an integration with on-chain components (e.g., pkg/onchain, pkg/contracts) which are used for deploying contracts and interacting with blockchain elements.
- Configuration & External Integration – The “config” directory holds JSON files for exchange pairs, pools, and symbol identification, indicating how the system is set up for different data sources.
- Container Orchestration – The existence of Dockerfiles in the “build” folder, a “docker-compose” directory, and a dedicated README section for Docker deployment indicates a strong container-based deployment model.
- CI/CD Pipelines – The .github/workflows directory suggests automated build and deployment processes which, while not directly part of the runtime architecture, are important in the development lifecycle.

##### Relationships & Data Flows
- External Data Sources: Represent the various exchanges (CEXs and DEXs) from which the scraper fetches real-time trade data.
- Data Collection: Following scraping, show that the data flows into a collector module that aggregates data from different scrapers (or even simulation inputs). This step might be represented as a separate “Collector” component.
- Supporting Modules: Include a segment for “Configuration” which supplies parameters (via JSON files) to the scraper and processor, and “Monitoring” to represent the metrics push via the Pushgateway (using the required environment variables).

##### Architectural Patterns & Design Principles
- Micro-Component Separation: Even though the project may be run as a single container/application, its structure shows clear separation of concerns (scraping, collection, processing, on-chain interaction).
- Modularity: The division into “pkg/”, “cmd/”, and “config/” indicates modular design and ease of extensibility for additional exchanges or filters.
- Containerization: Emphasize that the project is built for a containerized environment, simplifying deployment across various machines.
- CI/CD Integration: Although not part of runtime, the inclusion of GitHub workflows shows automated testing and deployment pipelines.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Components
    exchanges["External Crypto Exchanges"]:::external

    %% Deployment Components
    cicd["CI/CD Pipelines"]:::cicd
    orchestrate["Container Orchestration"]:::orchestration

    %% Docker Container Boundary
    subgraph "Docker Container"
        cfg["Configuration & External Integration"]:::config
        ds["Data Scraper"]:::module
        sim["Data Simulator"]:::module
        coll["Collector & Processor"]:::module
        sc["Smart Contract Interface"]:::module
    end

    %% Data Flow Arrows
    exchanges -->|"dataFetching"| ds
    cfg -->|"config"| ds
    cfg -->|"config"| coll
    sim -->|"simulationInput"| coll
    ds -->|"dataAggregation"| coll
    coll -->|"publishOnChain"| sc

    %% Deployment Flow
    cicd -->|"automatedDeploy"| orchestrate
    orchestrate -->|"deployContainer"| ds

    %% Click Events for Component Mapping

    %% Data Scraper
    click ds "https://github.com/diadata-org/decentral-feeder/tree/master/cmd/scrapers"
    click ds "https://github.com/diadata-org/decentral-feeder/tree/master/pkg/scrapers"

    %% Data Simulator
    click sim "https://github.com/diadata-org/decentral-feeder/tree/master/cmd/simulators"
    click sim "https://github.com/diadata-org/decentral-feeder/tree/master/pkg/simulations"

    %% Collector & Processor
    click coll "https://github.com/diadata-org/decentral-feeder/tree/master/pkg/processor"
    click coll "https://github.com/diadata-org/decentral-feeder/tree/master/pkg/metafilters"
    click coll "https://github.com/diadata-org/decentral-feeder/tree/master/pkg/simulations/simulationProcessor"

    %% Smart Contract Interface
    click sc "https://github.com/diadata-org/decentral-feeder/tree/master/pkg/onchain"
    click sc "https://github.com/diadata-org/decentral-feeder/tree/master/pkg/contracts"

    %% Configuration & External Integration
    click cfg "https://github.com/diadata-org/decentral-feeder/tree/master/config"
    click cfg "https://github.com/diadata-org/decentral-feeder/tree/master/config/pairs"
    click cfg "https://github.com/diadata-org/decentral-feeder/tree/master/config/pools"
    click cfg "https://github.com/diadata-org/decentral-feeder/tree/master/config/symbolIdentification"

    %% Container Orchestration
    click orchestrate "https://github.com/diadata-org/decentral-feeder/tree/master/build"
    click orchestrate "https://github.com/diadata-org/decentral-feeder/blob/master/docker-compose/docker-compose.yaml"

    %% CI/CD Pipelines
    click cicd "https://github.com/diadata-org/decentral-feeder/tree/master/.github/workflows"

    %% Style Definitions
    classDef external fill:#A2D9CE,stroke:#2471A3,stroke-width:2px
    classDef module fill:#AED6F1,stroke:#3498DB,stroke-width:2px
    classDef config fill:#F9E79F,stroke:#F1C40F,stroke-width:2px
    classDef orchestration fill:#D7BDE2,stroke:#8E44AD,stroke-width:2px
    classDef cicd fill:#F5B7B1,stroke:#E74C3C,stroke-width:2px
```
