# opendatadiscovery/odd-collectors architecture

> Architecture diagram of the GitHub repository opendatadiscovery/odd-collectors, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opendatadiscovery/odd-collectors
- Repository: https://github.com/opendatadiscovery/odd-collectors

## Overview

The OpenDataDiscovery Collectors project is essentially a multi-variant daemon-style service designed to periodically load metadata from many different data sources through a common “adapter” abstraction. There are several collectors present—one generic (odd-collector) and several cloud-specific versions (odd-collector-aws, odd-collector-azure, odd-collector-gcp)—as well as an SDK (odd-collector-sdk) that appears to be used for exposing common API functionality.

##### Main Components
- Collector Daemon: The central engine that runs as a daemon, periodically processing jobs. It loads the collector configuration (via the collector_config.yaml file) and manages the scheduling of metadata extraction.
- Configuration Manager: A component responsible for loading configuration from multiple sources (the local YAML file, environment variables, and potentially a Secrets Backend such as AWS Systems Manager Parameter Store). Emphasize priority order here.
- Adapters Abstraction: Each collector embeds dozens of adapters, each of which is responsible to interface with a specific type of data source (databases, cloud services, BI tools, etc.). These adapters encapsulate the logic to fetch metadata without reading actual data.
- Plugin Manager: The collector reads a “plugins” list (each plugin representing a specific adapter configuration) and instantiates the correct adapter for that data source.
- External Data Sources: Represent the various metadata sources (e.g., Athena, DynamoDB, Azure SQL, BigQuery, etc.) that the adapters connect to.
- ODD Platform Integration: A component or API client within the collector responsible for sending the collected metadata to the Open Data Discovery platform.
- Secrets Backend Integration: An optional but critical part that retrieves configuration options (like credentials and connection parameters) from backends such as AWS SSM.

- That the Scheduler triggers the Collector’s execution loop, leading to each plugin (and its adapter) performing a metadata extraction job before sending the result to the ODD Platform over an API connection.

##### Architectural Patterns and Technologies
- “plugin-based” architecture: each adapter operates independently and may follow similar internal patterns, such as parsing metadata, transforming it via a grammar parser (as seen in many adapter-specific subdirectories), and then relaying structured data.
- Mention the separation of concerns illustrated by having distinct directories for domain logic, error handling, configuration, web/shell entry points (e.g., __main__.py), and tests.
- Optionally, note the usage of secrets management and automated release (via tagging and GitHub workflows).

- Clear Labels: Each block should have a name (e.g., “Configuration Loader”, “Plugin Manager”, “Athena Adapter”, “Secrets Backend (AWS SSM)”, etc.).
- That each variant (generic, AWS, Azure, GCP) might be considered a “module” or “service” with similar internal architectures but different implementations of the Adapters layer.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources Subgraph
    subgraph "External Data Sources"
        ext_generic["Generic Data Sources"] 
        ext_aws["AWS Data Sources"]
        ext_azure["Azure Data Sources"]
        ext_gcp["GCP Data Sources"]
    end

    %% Adapters Layer Subgraph
    subgraph "Adapters Layer"
        adapter_generic["Generic Adapters"]
        adapter_aws["AWS Adapters"]
        adapter_azure["Azure Adapters"]
        adapter_gcp["GCP Adapters"]
        adapter_loader["Adapter Loader"]
    end

    %% Configuration & Secrets Subgraph
    subgraph "Configuration & Secrets"
        config_manager["Configuration Manager"]
        secrets_backend["Secrets Backend"]
    end

    %% Collector Core Subgraph
    subgraph "Collector Core"
        daemon_generic["Collector Daemon (Generic)"]
        daemon_aws["Collector Daemon (AWS)"]
        daemon_azure["Collector Daemon (Azure)"]
        daemon_gcp["Collector Daemon (GCP)"]
        scheduler["Scheduler"]
        plugin_manager["Plugin Manager"]
        api_client["ODD Platform API Client"]
    end

    %% External Data Sources to Adapters
    ext_generic -->|"config"| adapter_generic
    ext_aws -->|"config"| adapter_aws
    ext_azure -->|"config"| adapter_azure
    ext_gcp -->|"config"| adapter_gcp

    %% Plugin Manager and Adapters Interaction via Adapter Loader
    plugin_manager -->|"instantiate_adapter"| adapter_loader
    adapter_loader --> adapter_generic
    adapter_loader --> adapter_aws
    adapter_loader --> adapter_azure
    adapter_loader --> adapter_gcp

    %% Configuration Flow
    config_manager -->|"load_config"| plugin_manager
    secrets_backend -->|"provide_secrets"| config_manager

    %% Scheduling and Execution Flow
    scheduler -->|"trigger_plugins"| plugin_manager
    daemon_generic --> plugin_manager
    daemon_aws --> plugin_manager
    daemon_azure --> plugin_manager
    daemon_gcp --> plugin_manager

    %% Metadata sending
    plugin_manager -->|"send_metadata"| api_client

    %% Click Events for Collector Daemons
    click daemon_generic "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector/odd_collector/__main__.py"
    click daemon_aws "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-aws/odd_collector_aws/__main__.py"
    click daemon_azure "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-azure/odd_collector_azure/__main__.py"
    click daemon_gcp "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-gcp/odd_collector_gcp/__main__.py"

    %% Click Events for Configuration Manager
    click config_manager "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector/collector_config.yaml"
    click config_manager "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-sdk/odd_collector_sdk/domain/collector_config_loader.py"

    %% Click Events for Adapters
    click adapter_generic "https://github.com/opendatadiscovery/odd-collectors/tree/main/odd-collector/odd_collector/adapters/"
    click adapter_aws "https://github.com/opendatadiscovery/odd-collectors/tree/main/odd-collector-aws/odd_collector_aws/adapters/"
    click adapter_azure "https://github.com/opendatadiscovery/odd-collectors/tree/main/odd-collector-azure/odd_collector_azure/adapters/"
    click adapter_gcp "https://github.com/opendatadiscovery/odd-collectors/tree/main/odd-collector-gcp/odd_collector_gcp/adapters/"
    click adapter_loader "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-sdk/odd_collector_sdk/load_adapter.py"

    %% Click Events for Plugin Manager
    click plugin_manager "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector/odd_collector/domain/plugin.py"
    click plugin_manager "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-sdk/odd_collector_sdk/domain/plugin.py"

    %% Click Events for External Data Sources
    click ext_generic "https://github.com/opendatadiscovery/odd-collectors/tree/main/odd-collector/config_examples/"
    click ext_aws "https://github.com/opendatadiscovery/odd-collectors/tree/main/odd-collector-aws/config_examples/"
    click ext_azure "https://github.com/opendatadiscovery/odd-collectors/tree/main/odd-collector-azure/config_examples/"
    click ext_gcp "https://github.com/opendatadiscovery/odd-collectors/tree/main/odd-collector-gcp/config_examples/"

    %% Click Event for ODD Platform API Client
    click api_client "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-sdk/odd_collector_sdk/api/datasource_api.py"

    %% Click Events for Secrets Backend
    click secrets_backend "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-sdk/odd_collector_sdk/secrets/aws/ssm_parameter_store.py"
    click secrets_backend "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-sdk/odd_collector_sdk/secrets/base_secrets.py"
    click secrets_backend "https://github.com/opendatadiscovery/odd-collectors/blob/main/odd-collector-sdk/odd_collector_sdk/secrets/secrets_backend_factory.py"

    %% Styles
    classDef external fill:#ffe6cc,stroke:#cc6600,stroke-width:2px;
    class ext_generic,ext_aws,ext_azure,ext_gcp external;
    
    classDef adapter fill:#cce5ff,stroke:#3366cc,stroke-width:2px;
    class adapter_generic,adapter_aws,adapter_azure,adapter_gcp,adapter_loader adapter;
    
    classDef config fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    class config_manager config;
    
    classDef secrets fill:#d9ead3,stroke:#38761d,stroke-width:2px;
    class secrets_backend secrets;
    
    classDef daemon fill:#ccffcc,stroke:#339900,stroke-width:2px;
    class daemon_generic,daemon_aws,daemon_azure,daemon_gcp daemon;
    
    classDef scheduler fill:#ffffcc,stroke:#999900,stroke-width:2px;
    class scheduler scheduler;
    
    classDef plugin fill:#e6e6e6,stroke:#666666,stroke-width:2px;
    class plugin_manager plugin;
    
    classDef api fill:#e6ccff,stroke:#993399,stroke-width:2px;
    class api_client api;
```
