How eclipse-edc/samples works
Project Overview and Purpose
• The repository is a collection of sample implementations for the Eclipse Dataspace Components (EDC) framework. It is aimed at onboarding new community members by demonstrating various scopes (Basic, Transfer, Advanced, Policy, and Federated Catalog) so that users learn how to work with connectors, extensions, and integrations (such as telemetry, streaming, policy enforcement, etc.).
• The samples cover different aspects of the EDC framework – from the simplest connector implementations in the “basic” scope to complex integrations like custom runtimes, federated catalogs, Kafka streaming, and policy enforcement in other scopes.
Analyzing the File Structure
• Top-Level Organization: You see directories like “basic”, “advanced”, “transfer”, “policy”, “federated-catalog”, “system-tests”, and “util”. Each directory groups samples focusing on a specific EDC use case.
• Within each sample module:
• Gradle build files (build.gradle.kts) for dependency and build management.
• Configuration files (e.g., config.properties, JSON files for datasets, contract definitions, policies etc.) that represent runtime configurations.
• For the “advanced” and “transfer” samples, the presence of Dockerfiles, docker-compose files, and integration test resources indicates containerized runtimes and external service integrations such as Prometheus, OpenTelemetry, and Kafka.
Identifying Main Components
• Sample Scopes:
• Basic: Illustrates fundamental connector setup, health endpoints, and configuration examples.
• Transfer: Focuses on data transfer processes including negotiation, provider push, consumer pull, event-driven transfers, and streaming mechanisms.
• Advanced: Showcases custom runtime examples (like OpenTelemetry integration) and additional runtime configurations.
• Policy: Focuses on policy evaluation and enforcement (with sample policy functions and constraints).
• Federated Catalog: Demonstrates how catalog connectors and node resolvers work, including embedded and standalone deployments.
• Core runtime component:
• Each sample implements and runs on the EDC runtime, so include an “EDC Runtime” layer representing the container that loads extensions via Java’s SPI mechanism (visible in META-INF/services).
• External Services and Integrations:
• Messaging or Streaming: Kafka (for streaming data scenarios in transfer samples).
• Monitoring & Observability: OpenTelemetry and Prometheus (in advanced samples with Docker/Docker Compose setups).
• HTTP/REST APIs: Health endpoints, data transfer endpoints, and any custom API controllers.
• Test Infrastructure:
• System Tests: Represent integration tests (found under system-tests) that simulate end-to-end flows across the different scopes.
Mapping Relationships and Data Flow
• How client requests flow from a REST API (for example, the Health API Controller in the basic scope) into the EDC runtime.
• How configuration files (e.g., config.properties, JSON definitions) feed into the runtime to configure connectors, contracts, assets, and policies.
• How sample modules interact via the extension mechanism – each sample registers its own ServiceExtension component.
• In federated catalog samples, demonstrate how catalog queries are propagated between connectors and a separate node resolver which crawls multiple catalogs.
Architectural Patterns and Design Principles
• Modular design: Each scope is an independent module, enabling learning and extensibility.
• Plugin/Extension Architecture: Emphasize the use of Java SPI (via META-INF/services) that allows dynamic loading of service extensions.
• Separation of Concerns: Illustrate the distinction between connector logic, policy enforcement, configuration management, and transfer orchestration.
• Containerization & External Integration: Show how Docker and Docker Compose are used (especially in advanced samples) to simulate a multi-container environment with monitoring (Prometheus) and distributed tracing (OpenTelemetry).