How novonordisk-opensource/connector.databricks works
Project type and purpose
• This is an R package (“connector.databricks”) that extends the generic {connector} interface to support Databricks volumes (via REST API) and tables (via ODBC).
• It provides R6‐style connectors, utility functions, and methods for listing, reading, writing, and deleting objects in Databricks.
File structure
• R/: core source code
• DatabricksClient.R — handles authentication and HTTP calls to the Databricks REST API.
• table*.R — defines the table connector, methods for read/write, and utilities.
• volume*.R — defines the volume connector, methods for read/write, and volume‐specific utils.
• utils.R, connector_imports.R, working_connection.R — shared helpers and setup.
• inst/config/example_yaml.yaml — example config for integration with {connector}.
• man/: auto‐generated documentation for each function and class.
• tests/: unit tests for client, table utils, volume API, and logging.
• .github/actions & workflows — CI for linting, testing.
• pkgdown/: static website configuration.
README for insights
• Two main connector “types”: tables (via odbc::databricks()) and volumes (via DatabricksClient + ~/.databrickscfg).
• Standard methods: list_content_cnt(), read_cnt(), write_cnt(), remove_cnt().
• Integration with {connector} package and YAML‐based configuration.
Main components
a. User/R Session
• R script or interactive session calling connector.databricks API.
b. connector.databricks Package
• Connector Factory: connector_databricks_table() & connector_databricks_volume().
• DatabricksClient (REST API wrapper).
• TableModule (ODBC interaction).
• VolumeModule (HTTP endpoints).
• Shared utilities (serialization, logging).
c. External Dependencies
• odbc::databricks() driver (ODBC).
• Databricks REST API endpoints (Volumes).
• Environment/config (DATABRICKS_CONFIG_FILE, ~/.databrickscfg).
d. Data Stores/Services
• Databricks Cluster — Catalog, Schema, Tables.
• Databricks File/Volume storage (DBFS or mounted volumes).
e. Optional Integration
• {connector} core package (if used).
Relationships & data flows
• The user calls connector_databricks_table() → returns an R6 object with methods that internally call odbc::DBI to query Databricks tables.
• The user calls connector_databricks_volume() → returns an R6 object that calls DatabricksClient$api() to perform volume operations.
• All read_cnt/write_cnt/remove_cnt calls flow through shared utils (serialization, logging), then out to ODBC or HTTP.
• Config file or environment variable is loaded by DatabricksClient at instantiation.
Architectural patterns & design principles
• Adapter pattern: wraps ODBC and REST API to present a uniform {connector} interface.
• R6 classes for connector instances encapsulate state (catalog, schema, path).
• Separation of concerns: client vs. table vs. volume modules.
• Dependency injection: under‐the‐hood reliance on odbc package and httr/glue for REST.