# diadata-org/diadata architecture

> Architecture diagram of the GitHub repository diadata-org/diadata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/diadata-org/diadata
- Repository: https://github.com/diadata-org/diadata

## Overview

##### Major Components

a. Data Sources and External Feeds:
- Many external data origins (on-chain sources, centralized exchange APIs, DEXs, DeFi platforms, NFT markets, metaverse data etc.). Include an expanded set of “Additional Data Sources” that highlights future or emerging sources.

b. Data Ingestion Layer:
- Diverse set of scrapers and collectors found under the cmd folders (for example, exchange scrapers, blockchain scrapers, historical scrapers, liquidity scrapers, and foreign scrapers).
- Them as parallel microservices that continuously poll or receive data.
- That each scraper is configured (using JSON, Go modules, or Dockerfiles) to process its respective data feed.

c. Data Processing and Aggregation:
- Auxiliary helper components (for example, risk metrics, MAIR methodologies) that are integrated in the processing flow.

d. Data Storage and Messaging:
- Since the project uses database configuration files (InfluxDB, PostgreSQL, Redis) and even Kafka in CI/CD pipelines (e.g., Tekton tasks), include storage/messaging clusters.
- Connections from the processing layer to these systems.
- With common symbols for databases or messaging brokers.

e. Data Delivery/Distribution Layer:
- Two delivery paradigms:
- On-chain delivery (DIA xNode) where processed data is pushed via oracle smart contracts.
- Off-chain delivery (DIA xBase) via REST API, GraphQL endpoints (see components under cmd/http/ and documentation references).
- API gateways and/or smart contract endpoints that interface with dApps.

f. Infrastructure and Deployment:
- On a separate section, capture deployment artifacts: the Dockerfiles, docker-compose configurations, Tekton pipelines (in .tekton and deployments/k8s-yaml), GitHub workflows, and other CI/CD scripts.
- This section demonstrates how the entire system is containerized, orchestrated, and continuously deployed.

##### Relationships and Flows

- Scrapers to the central aggregation/processing engine.
- From aggregation, direct flows toward data storage (databases, caches) and messaging services.

##### Architectural Patterns and Design Principles

- Microservices approach—the system is broken into many discrete small services (each in its own cmd subdirectory) interacting over defined interfaces.
- Use of containerization (multiple Dockerfiles and Kubernetes/tekton deployments) to ensure scalability and reliability.
- Acknowledge the separation of concerns: data collection, processing, and delivery are modularized and decoupled for easier maintainability and extensibility.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Define class styles
    classDef external fill:#85C1E9,stroke:#1F618D,stroke-width:2px;
    classDef ingestion fill:#A9DFBF,stroke:#1D8348,stroke-width:2px;
    classDef processing fill:#82E0AA,stroke:#145A32,stroke-width:2px;
    classDef storage fill:#F5B041,stroke:#AF601A,stroke-width:2px;
    classDef delivery fill:#BB8FCE,stroke:#6C3483,stroke-width:2px;
    classDef deployment fill:#D5D8DC,stroke:#566573,stroke-width:2px;

    %% Data Sources & External Feeds Subgraph
    subgraph "Data Sources & External Feeds"
        E1["Exchange APIs"]
        E2["On-chain Data Streams"]
        E3["DEXs & DeFi Platforms"]
        E4["NFT Markets"]
        E5["Metaverse Data"]
        E6["Additional Data Sources"]
        E7["Community Contributions"]
    end
    class E1,E2,E3,E4,E5,E6,E7 external;

    %% Data Ingestion Layer Subgraph
    subgraph "Data Ingestion Layer"
        I1["Exchange Scraper Service"]
        I2["Blockchain Scraper Service"]
        I3["Foreign Scraper Service"]
        I4["Liquidity Scraper Service"]
        I5["Stock Scraper Service"]
        I6["Ratescraper Service"]
    end
    class I1,I2,I3,I4,I5,I6 ingestion;

    %% Data Processing & Aggregation Subgraph
    subgraph "Data Processing & Aggregation"
        P1["Filters & Validators"]
        P2["Data Aggregator"]
        P3["Additional Processing Services"]
    end
    class P1,P2,P3 processing;

    %% Data Storage & Messaging Subgraph
    subgraph "Data Storage & Messaging"
        S1["InfluxDB Cluster"]
        S2["PostgreSQL Cluster"]
        S3["Redis Cache"]
        S4["Kafka Messaging System"]
    end
    class S1,S2,S3,S4 storage;

    %% Data Delivery/Distribution Subgraph
    subgraph "Data Delivery/Distribution"
        D1["REST API Gateway"]
        D2["GraphQL Server"]
        D3["Oracle Smart Contract Interface"]
    end
    class D1,D2,D3 delivery;

    %% Infrastructure & Deployment Subgraph
    subgraph "Infrastructure & Deployment"
        F1["Docker & Containerization"]
        F2["CI/CD Pipelines (.tekton)"]
        F3["Deployments (docker-compose/k8s)"]
        F4["GitHub Workflows"]
    end
    class F1,F2,F3,F4 deployment;

    %% Data flow: External Sources to Ingestion
    E1 --> I1
    E2 --> I2
    E3 --> I1
    E3 --> I2
    E4 --> I3
    E5 --> I4
    E6 --> I1
    E6 --> I2
    E6 --> I3
    E6 --> I4
    E6 --> I5
    E6 --> I6
    E7 --> I1

    %% Data flow: Ingestion to Processing
    I1 --> P2
    I2 --> P2
    I3 --> P1
    I4 --> P3
    I5 --> P3
    I6 --> P2

    %% Data flow: Processing to Storage & Messaging
    P1 --> S1
    P2 --> S2
    P3 --> S3
    P2 --> S4

    %% Data flow: Processing to Delivery
    P2 --> D1
    P2 --> D2
    P2 --> D3

    %% Feedback loop from Delivery back to Processing
    D1 -- "feedback" --> P1
    D2 -- "feedback" --> P1

    %% Infrastructure support flows
    F1 --> I1
    F1 --> I2
    F1 --> I3
    F1 --> I4
    F1 --> I5
    F1 --> I6
    F1 --> D1
    F1 --> D2
    F1 --> D3
    F1 --> P1
    F1 --> P2
    F1 --> P3
    F3 --> S1
    F3 --> S2
    F3 --> S3
    F3 --> S4
    F2 --> F1
    F3 --> F1
    F4 --> F2

    %% Click Events for Data Sources
    click E1 "https://github.com/diadata-org/diadata/tree/master/config/exchanges"
    click E2 "https://github.com/diadata-org/diadata/tree/master/config/blockchains"
    click E3 "https://github.com/diadata-org/diadata/tree/master/config/uniswap"
    click E4 "https://github.com/diadata-org/diadata/tree/master/config/nftContracts"
    click E6 "https://github.com/diadata-org/diadata/tree/master/documentation/data-sources"

    %% Click Events for Data Ingestion Layer
    click I1 "https://github.com/diadata-org/diadata/tree/master/cmd/exchange-scrapers"
    click I2 "https://github.com/diadata-org/diadata/tree/master/cmd/blockscraper"
    click I3 "https://github.com/diadata-org/diadata/tree/master/cmd/foreignscraper"
    click I4 "https://github.com/diadata-org/diadata/tree/master/cmd/liquidityScraper"
    click I5 "https://github.com/diadata-org/diadata/tree/master/cmd/stock-scrapers"
    click I6 "https://github.com/diadata-org/diadata/tree/master/cmd/ratescrapers"

    %% Click Events for Data Processing & Aggregation
    click P1 "https://github.com/diadata-org/diadata/tree/master/internal/pkg/filtersBlockService"
    click P2 "https://github.com/diadata-org/diadata/tree/master/pkg/dia"
    click P3 "https://github.com/diadata-org/diadata/tree/master/cmd/services"

    %% Click Events for Data Storage & Messaging
    click S1 "https://github.com/diadata-org/diadata/blob/master/config/Influx.json"
    click S2 "https://github.com/diadata-org/diadata/blob/master/deployments/docker-compose.postgres.yml"
    click S3 "https://github.com/diadata-org/diadata/blob/master/deployments/docker-compose.redis.yml"
    click S4 "https://github.com/diadata-org/diadata/blob/master/.tekton/PipelineRun/PipelineRun.Build.Deploy.TektonCI.assetCollectionService.yaml"

    %% Click Events for Data Delivery/Distribution
    click D1 "https://github.com/diadata-org/diadata/tree/master/cmd/http/restServer"
    click D2 "https://github.com/diadata-org/diadata/tree/master/cmd/http/graphqlServer"
    click D3 "https://github.com/diadata-org/diadata/tree/master/cmd/oraclebuilder"

    %% Click Events for Infrastructure & Deployment
    click F1 "https://github.com/diadata-org/diadata/tree/master/build"
    click F2 "https://github.com/diadata-org/diadata/blob/master/.tekton"
    click F3 "https://github.com/diadata-org/diadata/tree/master/deployments"
    click F4 "https://github.com/diadata-org/diadata/tree/master/.github/workflows"
```
