How hasura/ndc-hub works
Project Type and Purpose
• The project is a platform consisting of a registry of data connectors along with the necessary tooling and SDK components to build, test, and deploy these connectors.
• The README explains that the primary aim is to let Hasura users quickly obtain a GraphQL API on any data source via prebuilt or custom connectors. It also emphasizes end-to-end (e2e) testing, packaging, and connector metadata management.
File Structure
• The top-level directories indicate multiple facets of the platform:
• .github/workflows: Contains CI/CD configurations and tests (such as go-tests, e2e tests, metadata schema validations, etc).
• connector-metadata-types: Contains TypeScript files (e.g. connectorTypes.ts, schemaGenerator.ts, and index.ts) which define the schema, types, and functions used for creating connector metadata. These files include several helper functions to generate or validate connector schemas.
• registry-automation: Implements backend utilities in Go (e.g. in the cmd and pkg folders) to run CI, validation, e2e tests, and register connector metadata to the hub. It also includes several function files (like validate.go, utils.go, and others) to perform tasks such as schema validation, connector packaging, and integration testing.
• registry: Organized in a way that each subdirectory (hasura/*) represents connectors for different data sources. This part includes documentation, metadata JSONs, release folders, and test configurations.
• rfcs and scripts: Contain design documents for deployment, packaging, and automation, plus scripting utilities (e.g., update-ndc-postgres.sh).
Key Components and Their Relationships
a. Connector Metadata Module:
• Component: Connector Metadata Types (connector-metadata-types)
• Contains: TypeScript source files that define connector types, generate JSON schemas, and package connector metadata.
• Functions: Schema generators and connector type validators; these functions are central to ensuring that the metadata follows the required format.
b. Registry Automation:
• Component: Registry Automation (registry-automation)
• Contains: CLI commands (in the cmd folder), internal packages (pkg/ndchub and pkg/validate), and various utilities.
• Functions: Utility functions such as validate.go and utils.go that check metadata consistency and drive CI workflows, as well as functions to invoke end-to-end tests.
c. Connector Registry:
• Component: Registry (registry)
• Contains: Individual directories for each connector (like hasura/mysql, hasura/elasticsearch, etc.) with their metadata, release information, and e2e test configurations.
• Relationship: The registry is updated/managed by the automated tools and CLI commands from the registry automation module.
d. CI/CD Configuration and Testing:
• Component: GitHub Actions Workflows (under .github/workflows)
• Contains: YAML files for executing tests, build pipelines, metadata validations, and deployments.
• Relationship: These workflows trigger both the metadata generation functions and registry automation functions.
e. RFCs and Additional Scripts:
• Component: RFCs and Scripts
• Contains: Documentation and guideline resources regarding deployment and packaging strategies.
• Relationship: They guide how the functions and processes in the application should behave and evolve over time.
Incorporating Functions
• For instance, inside the Connector Metadata Module, list functions for “schemaGenerator” and “index” to indicate how metadata is produced.
Illustrating Architectural Patterns and Technologies
• Separation of concerns: the metadata generation (TypeScript-based) is distinct from the automation and CLI (Go-based), and both eventually feed into the registry system.
• Directional data flow:
• Data flows from source (connector code and metadata) through the metadata types/functions.
• Then into the registry automation functions that validate, package, and test the metadata.
• Finally, the processed and released connectors get stored under the registry folder with all associated release information.
• Incorporate CI/CD pipeline as an orchestrator that triggers these functions.