How diadata-org/diadata works
Major Components
a. Data Sources and External Feeds:
• Many external data origins (on-chain sources, centralized exchange APIs, DEXs, DeFi platforms, NFT markets, metaverse data etc.). Include an expanded set of “Additional Data Sources” that highlights future or emerging sources.
b. Data Ingestion Layer:
• Diverse set of scrapers and collectors found under the cmd folders (for example, exchange scrapers, blockchain scrapers, historical scrapers, liquidity scrapers, and foreign scrapers).
• Them as parallel microservices that continuously poll or receive data.
• That each scraper is configured (using JSON, Go modules, or Dockerfiles) to process its respective data feed.
c. Data Processing and Aggregation:
• Auxiliary helper components (for example, risk metrics, MAIR methodologies) that are integrated in the processing flow.
d. Data Storage and Messaging:
• Since the project uses database configuration files (InfluxDB, PostgreSQL, Redis) and even Kafka in CI/CD pipelines (e.g., Tekton tasks), include storage/messaging clusters.
• Connections from the processing layer to these systems.
• With common symbols for databases or messaging brokers.
e. Data Delivery/Distribution Layer:
• Two delivery paradigms:
• On-chain delivery (DIA xNode) where processed data is pushed via oracle smart contracts.
• Off-chain delivery (DIA xBase) via REST API, GraphQL endpoints (see components under cmd/http/ and documentation references).
• API gateways and/or smart contract endpoints that interface with dApps.
f. Infrastructure and Deployment:
• On a separate section, capture deployment artifacts: the Dockerfiles, docker-compose configurations, Tekton pipelines (in .tekton and deployments/k8s-yaml), GitHub workflows, and other CI/CD scripts.
• This section demonstrates how the entire system is containerized, orchestrated, and continuously deployed.
Relationships and Flows
• Scrapers to the central aggregation/processing engine.
• From aggregation, direct flows toward data storage (databases, caches) and messaging services.
Architectural Patterns and Design Principles
• Microservices approach—the system is broken into many discrete small services (each in its own cmd subdirectory) interacting over defined interfaces.
• Use of containerization (multiple Dockerfiles and Kubernetes/tekton deployments) to ensure scalability and reliability.
• Acknowledge the separation of concerns: data collection, processing, and delivery are modularized and decoupled for easier maintainability and extensibility.