# owox/js-data-connectors architecture

> Architecture diagram of the GitHub repository owox/js-data-connectors, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/owox/js-data-connectors
- Repository: https://github.com/owox/js-data-connectors

## Overview

##### Main Components
- Core Module – Located under “src/Core”, this folder contains abstract classes (AbstractConfig.js, AbstractConnector.js, AbstractPipeline.js, AbstractStorage.js) and fundamental implementations (like GoogleSheetsConfig.js and GoogleSheetsStorage.js) that define the interfaces and default behaviors.
- Integrations – Under “src/Integrations”, each subdirectory (such as BankOfCanada, FacebookMarketing, OpenExchangeRates, OpenHolidays, etc.) is a self-contained connector. These folders include specific implementations for connectors, pipelines (data processing flows), and sometimes custom configuration and credentials documentation – all built on top of the Core abstractions.
- Templates – “src/Templates” offers example or boilerplate code (like PublicEndPoint) that can be used to create new connectors easily.
- Resources and Metadata – “res” on the top level and in integration folders provide graphical assets and documentation that outline branding or guides.
- Deployment/Configuration Files – Files like “.clasp.json”, “.claspignore”, and “appsscript.json” play an important role in deploying these apps to Google Apps Script.

##### Relationships and Interactions
- Data Connector Interfaces: The abstract classes in the Core (for configuration, connectors, pipelines, and storage) dictate how a new integration must be structured. Each integration folder implements these interfaces.
- Modular Extensibility: Emphasize that adding a new data source simply involves creating a new module under “src/Integrations” that conforms to the Core abstractions.

##### Architectural Patterns and Design Principles
- Modular/Plugin Architecture: The clear separation between Core functionality and individual Integrations emphasizes a plug-in–like architecture.
- Abstraction and Inheritance: The use of abstract classes (AbstractConnector, AbstractPipeline, etc.) allows for consistent extension and customization.
- Serverless Deployment: The design is meant for Apps Script “serverless” environments. No standalone server or cloud orchestration is required – the logic runs directly on the cloud environment provided by Google.
- Separation of Concerns: By separating configuration, connection logic, data pipeline, and storage, the design adheres to the principles of clear, modular design.

##### Relevant Technologies/Frameworks
- JavaScript and Google Apps Script – the primary programming environment.
- CLASP (Command Line Apps Script Projects) – indicated by the .clasp.json and .claspignore files in several integration folders.
- Google Sheets and possibly Google BigQuery – as target data storages for imported data.
- GitHub as the code hosting and collaboration tool, along with standard open-source documents (CONTRIBUTING.md, LICENSE, README.md).

##### Final Reminders for the Principal Software Engineer
- Be detailed – capture the major abstractions and how each integration ties back to the core.
- Tailor the level of detail to your audience – while some aspects can be overviewed, the core and integration patterns should be convincingly detailed.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data Sources
    External_Bank["Bank of Canada API"]:::external
    External_Facebook["Facebook API"]:::external
    External_Exchange["Exchange Rates API"]:::external
    External_Holidays["Holidays API"]:::external

    %% Integrations Subgraph
    subgraph "Integrations"
        BankConnector["Bank of Canada Connector"]:::integration
        BankPipeline["Bank of Canada Pipeline"]:::integration

        FacebookConnector["Facebook Marketing Connector"]:::integration
        FacebookPipeline["Facebook Marketing Pipeline"]:::integration

        OpenExConnector["Open Exchange Rates Connector"]:::integration
        OpenExPipeline["Open Exchange Rates Pipeline"]:::integration

        OpenHolidaysConnector["Open Holidays Connector"]:::integration
        OpenHolidaysPipeline["Open Holidays Pipeline"]:::integration
    end

    %% Core Module Subgraph
    subgraph "Core Module"
        Core_Config["AbstractConfig & Config Implementation"]:::core
        Core_Connector["AbstractConnector & Connector Implementation"]:::core
        Core_Pipeline["AbstractPipeline Implementation"]:::core
        Core_Storage["AbstractStorage (GoogleSheetsStorage)"]:::core
    end

    %% Templates and Deployment/Configuration Files and Resources
    Templates["Templates: PublicEndPoint Boilerplate"]:::template
    Deployment["Deployment/Configuration Files"]:::config
    Resources["Global Resources & Metadata"]:::resource

    %% Data flow from External Sources to Integrations to Core Storage
    External_Bank -->|"calls"| BankConnector
    BankConnector -->|"processes"| BankPipeline
    BankPipeline -->|"storesData"| Core_Storage

    External_Facebook -->|"calls"| FacebookConnector
    FacebookConnector -->|"processes"| FacebookPipeline
    FacebookPipeline -->|"storesData"| Core_Storage

    External_Exchange -->|"calls"| OpenExConnector
    OpenExConnector -->|"processes"| OpenExPipeline
    OpenExPipeline -->|"storesData"| Core_Storage

    External_Holidays -->|"calls"| OpenHolidaysConnector
    OpenHolidaysConnector -->|"processes"| OpenHolidaysPipeline
    OpenHolidaysPipeline -->|"storesData"| Core_Storage

    %% Integrations depend on Core Abstractions (dashed arrows)
    BankConnector -.->|"uses"| Core_Connector
    BankConnector -.->|"uses"| Core_Config
    BankPipeline -.->|"extends"| Core_Pipeline

    FacebookConnector -.->|"uses"| Core_Connector
    FacebookConnector -.->|"uses"| Core_Config
    FacebookPipeline -.->|"extends"| Core_Pipeline

    OpenExConnector -.->|"uses"| Core_Connector
    OpenExConnector -.->|"uses"| Core_Config
    OpenExPipeline -.->|"extends"| Core_Pipeline

    OpenHolidaysConnector -.->|"uses"| Core_Connector
    OpenHolidaysConnector -.->|"uses"| Core_Config
    OpenHolidaysPipeline -.->|"extends"| Core_Pipeline

    %% Templates influence Integration Connectors
    Templates -->|"boilerplate"| BankConnector
    Templates -->|"boilerplate"| FacebookConnector
    Templates -->|"boilerplate"| OpenExConnector
    Templates -->|"boilerplate"| OpenHolidaysConnector

    %% Deployment applies to the whole project and Resources support modules
    Deployment -->|"deploys"| Core_Storage
    Deployment -->|"deploys"| BankConnector
    Deployment -->|"deploys"| FacebookConnector
    Deployment -->|"deploys"| OpenExConnector
    Deployment -->|"deploys"| OpenHolidaysConnector
    Deployment -->|"deploys"| Resources

    %% Click Events
    click Core_Config "https://github.com/owox/js-data-connectors/tree/main/src/Core"
    click BankConnector "https://github.com/owox/js-data-connectors/tree/main/src/Integrations/BankOfCanada"
    click FacebookConnector "https://github.com/owox/js-data-connectors/tree/main/src/Integrations/FacebookMarketing"
    click OpenExConnector "https://github.com/owox/js-data-connectors/tree/main/src/Integrations/OpenExchangeRates"
    click OpenHolidaysConnector "https://github.com/owox/js-data-connectors/tree/main/src/Integrations/OpenHolidays"
    click Templates "https://github.com/owox/js-data-connectors/tree/main/src/Templates/PublicEndPoint"
    click Deployment "https://github.com/owox/js-data-connectors/blob/main/.clasp.json"
    click Resources "https://github.com/owox/js-data-connectors/tree/main/res"

    %% Styles
    classDef core fill:#9ad1ff,stroke:#333,stroke-width:2px;
    classDef integration fill:#ffdd89,stroke:#333,stroke-width:2px;
    classDef external fill:#f9c7c7,stroke:#333,stroke-width:2px,stroke-dasharray:5,5;
    classDef template fill:#c7f9d1,stroke:#333,stroke-width:2px;
    classDef config fill:#d1c7f9,stroke:#333,stroke-width:2px;
    classDef resource fill:#f9e79f,stroke:#333,stroke-width:2px;
```
