# novonordisk-opensource/connector.databricks architecture

> Architecture diagram of the GitHub repository novonordisk-opensource/connector.databricks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/novonordisk-opensource/connector.databricks
- Repository: https://github.com/novonordisk-opensource/connector.databricks

## Overview

##### Project type and purpose
- This is an R package (“connector.databricks”) that extends the generic {connector} interface to support Databricks **volumes** (via REST API) and **tables** (via ODBC).
- It provides R6‐style connectors, utility functions, and methods for listing, reading, writing, and deleting objects in Databricks.

##### File structure
- R/: core source code
- DatabricksClient.R — handles authentication and HTTP calls to the Databricks REST API.
- table*.R — defines the table connector, methods for read/write, and utilities.
- volume*.R — defines the volume connector, methods for read/write, and volume‐specific utils.
- utils.R, connector_imports.R, working_connection.R — shared helpers and setup.
- inst/config/example_yaml.yaml — example config for integration with {connector}.
- man/: auto‐generated documentation for each function and class.
- tests/: unit tests for client, table utils, volume API, and logging.
- .github/actions & workflows — CI for linting, testing.
- pkgdown/: static website configuration.

##### README for insights
- Two main connector “types”: tables (via odbc::databricks()) and volumes (via DatabricksClient + ~/.databrickscfg).
- Standard methods: list_content_cnt(), read_cnt(), write_cnt(), remove_cnt().
- Integration with {connector} package and YAML‐based configuration.

##### Main components
a. User/R Session
- R script or interactive session calling connector.databricks API.
b. connector.databricks Package
- Connector Factory: connector_databricks_table() & connector_databricks_volume().
- DatabricksClient (REST API wrapper).
- TableModule (ODBC interaction).
- VolumeModule (HTTP endpoints).
- Shared utilities (serialization, logging).
c. External Dependencies
- odbc::databricks() driver (ODBC).
- Databricks REST API endpoints (Volumes).
- Environment/config (DATABRICKS_CONFIG_FILE, ~/.databrickscfg).
d. Data Stores/Services
- Databricks Cluster — Catalog, Schema, Tables.
- Databricks File/Volume storage (DBFS or mounted volumes).
e. Optional Integration
- {connector} core package (if used).

##### Relationships & data flows
- The user calls connector_databricks_table() → returns an R6 object with methods that internally call odbc::DBI to query Databricks tables.
- The user calls connector_databricks_volume() → returns an R6 object that calls DatabricksClient$api() to perform volume operations.
- All read_cnt/write_cnt/remove_cnt calls flow through shared utils (serialization, logging), then out to ODBC or HTTP.
- Config file or environment variable is loaded by DatabricksClient at instantiation.

##### Architectural patterns & design principles
- Adapter pattern: wraps ODBC and REST API to present a uniform {connector} interface.
- R6 classes for connector instances encapsulate state (catalog, schema, path).
- Separation of concerns: client vs. table vs. volume modules.
- Dependency injection: under‐the‐hood reliance on odbc package and httr/glue for REST.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    U["User / R Session"]:::external

    %% Optional Integration
    ConnectorCore["{connector} Core"]:::optional

    %% Connector.databricks Package
    subgraph "connector.databricks Package"
        direction TB
        Factory["Connector Factory\n(connector_databricks_table(),\nconnector_databricks_volume())"]:::internal
        Client["DatabricksClient\n(REST API wrapper)"]:::internal
        TableConn["Table Connector\n(ODBC methods)"]:::internal
        VolumeConn["Volume Connector\n(HTTP methods)"]:::internal
        Utils["Shared Utilities\n(serialization, logging,\nconnection mgmt)"]:::internal
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        ODBCDriver["ODBC::databricks() Driver"]:::external
        RESTAPI["Databricks REST API"]:::external
        ConfigFile["Example Config\n(inst/config/example_yaml.yaml)"]:::external
        ConfigEnv["Env Var (~/.databrickscfg)"]:::external
    end

    %% Databricks Services
    subgraph "Databricks Services"
        direction TB
        Cluster["Databricks Cluster\n(Tables)"]:::service
        Storage["Databricks Storage\n(Volumes)"]:::service
    end

    %% Tests
    subgraph "Unit Tests"
        direction TB
        TestTable["test-table.R"]:::test
        TestTableUtils["test-table-utils.R"]:::test
        TestVolume["test-volume.R"]:::test
        TestVolumeAPI["test-volumes_api.R"]:::test
        TestLogger["test-cnt_logger_databricks.R"]:::test
        HelperTable["helper-table.R"]:::test
        Setup["setup.R"]:::test
    end

    %% Connections
    U --> Factory
    U -.- ConnectorCore
    Factory -.- ConnectorCore
    Factory --> TableConn
    Factory --> VolumeConn

    TableConn -->|"list/read/write/delete"| Utils
    VolumeConn -->|"list/read/write/delete"| Utils

    TableConn --> ODBCDriver
    ODBCDriver -->|"SQL queries"| Cluster

    VolumeConn --> Client
    Client --> RESTAPI
    RESTAPI --> Storage

    ConfigFile --> Client
    ConfigEnv --> Client

    %% Tests dependencies
    TestTable --> TableConn
    TestTableUtils --> TableConn
    TestVolume --> VolumeConn
    TestVolumeAPI --> Client
    TestLogger --> Utils
    HelperTable --> TestTable
    Setup --> TestTable

    %% Click Events
    click Factory "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/table.R"
    click Factory "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/volume.R"
    click Client "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/DatabricksClient.R"
    click TableConn "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/table_methods.R"
    click TableConn "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/table_utils.R"
    click VolumeConn "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/volume_api.R"
    click VolumeConn "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/volume_methods.R"
    click Utils "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/utils.R"
    click Utils "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/connector_imports.R"
    click Utils "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/cnt_logger_databricks.R"
    click Utils "https://github.com/novonordisk-opensource/connector.databricks/blob/main/R/working_connection.R"
    click ConfigFile "https://github.com/novonordisk-opensource/connector.databricks/blob/main/inst/config/example_yaml.yaml"
    click TestTable "https://github.com/novonordisk-opensource/connector.databricks/blob/main/tests/testthat/test-table.R"
    click TestTableUtils "https://github.com/novonordisk-opensource/connector.databricks/blob/main/tests/testthat/test-table-utils.R"
    click TestVolume "https://github.com/novonordisk-opensource/connector.databricks/blob/main/tests/testthat/test-volume.R"
    click TestVolumeAPI "https://github.com/novonordisk-opensource/connector.databricks/blob/main/tests/testthat/test-volumes_api.R"
    click TestLogger "https://github.com/novonordisk-opensource/connector.databricks/blob/main/tests/testthat/test-cnt_logger_databricks.R"
    click HelperTable "https://github.com/novonordisk-opensource/connector.databricks/blob/main/tests/testthat/helper-table.R"
    click Setup "https://github.com/novonordisk-opensource/connector.databricks/blob/main/tests/testthat/setup.R"

    %% Styling
    classDef internal fill:#D6EAF8,stroke:#1B4F72,color:#1B4F72
    classDef external fill:#D5F5E3,stroke:#145A32,color:#145A32
    classDef service fill:#FAD7A0,stroke:#B9770E,color:#873600
    classDef optional fill:#FDEBD0,stroke-dasharray: 5 5,stroke:#7E5109,color:#7E5109
    classDef test fill:#F2F3F4,stroke:#566573,color:#566573
```
