How opendatadiscovery/odd-collectors works
The OpenDataDiscovery Collectors project is essentially a multi-variant daemon-style service designed to periodically load metadata from many different data sources through a common “adapter” abstraction. There are several collectors present—one generic (odd-collector) and several cloud-specific versions (odd-collector-aws, odd-collector-azure, odd-collector-gcp)—as well as an SDK (odd-collector-sdk) that appears to be used for exposing common API functionality.
Main Components
• Collector Daemon: The central engine that runs as a daemon, periodically processing jobs. It loads the collector configuration (via the collector_config.yaml file) and manages the scheduling of metadata extraction.
• Configuration Manager: A component responsible for loading configuration from multiple sources (the local YAML file, environment variables, and potentially a Secrets Backend such as AWS Systems Manager Parameter Store). Emphasize priority order here.
• Adapters Abstraction: Each collector embeds dozens of adapters, each of which is responsible to interface with a specific type of data source (databases, cloud services, BI tools, etc.). These adapters encapsulate the logic to fetch metadata without reading actual data.
• Plugin Manager: The collector reads a “plugins” list (each plugin representing a specific adapter configuration) and instantiates the correct adapter for that data source.
• External Data Sources: Represent the various metadata sources (e.g., Athena, DynamoDB, Azure SQL, BigQuery, etc.) that the adapters connect to.
• ODD Platform Integration: A component or API client within the collector responsible for sending the collected metadata to the Open Data Discovery platform.
• Secrets Backend Integration: An optional but critical part that retrieves configuration options (like credentials and connection parameters) from backends such as AWS SSM.
• That the Scheduler triggers the Collector’s execution loop, leading to each plugin (and its adapter) performing a metadata extraction job before sending the result to the ODD Platform over an API connection.
Architectural Patterns and Technologies
• “plugin-based” architecture: each adapter operates independently and may follow similar internal patterns, such as parsing metadata, transforming it via a grammar parser (as seen in many adapter-specific subdirectories), and then relaying structured data.
• Mention the separation of concerns illustrated by having distinct directories for domain logic, error handling, configuration, web/shell entry points (e.g., __main__.py), and tests.
• Optionally, note the usage of secrets management and automated release (via tagging and GitHub workflows).
• Clear Labels: Each block should have a name (e.g., “Configuration Loader”, “Plugin Manager”, “Athena Adapter”, “Secrets Backend (AWS SSM)”, etc.).
• That each variant (generic, AWS, Azure, GCP) might be considered a “module” or “service” with similar internal architectures but different implementations of the Adapters layer.