How owox/js-data-connectors works
Main Components
• Core Module – Located under “src/Core”, this folder contains abstract classes (AbstractConfig.js, AbstractConnector.js, AbstractPipeline.js, AbstractStorage.js) and fundamental implementations (like GoogleSheetsConfig.js and GoogleSheetsStorage.js) that define the interfaces and default behaviors.
• Integrations – Under “src/Integrations”, each subdirectory (such as BankOfCanada, FacebookMarketing, OpenExchangeRates, OpenHolidays, etc.) is a self-contained connector. These folders include specific implementations for connectors, pipelines (data processing flows), and sometimes custom configuration and credentials documentation – all built on top of the Core abstractions.
• Templates – “src/Templates” offers example or boilerplate code (like PublicEndPoint) that can be used to create new connectors easily.
• Resources and Metadata – “res” on the top level and in integration folders provide graphical assets and documentation that outline branding or guides.
• Deployment/Configuration Files – Files like “.clasp.json”, “.claspignore”, and “appsscript.json” play an important role in deploying these apps to Google Apps Script.
Relationships and Interactions
• Data Connector Interfaces: The abstract classes in the Core (for configuration, connectors, pipelines, and storage) dictate how a new integration must be structured. Each integration folder implements these interfaces.
• Modular Extensibility: Emphasize that adding a new data source simply involves creating a new module under “src/Integrations” that conforms to the Core abstractions.
Architectural Patterns and Design Principles
• Modular/Plugin Architecture: The clear separation between Core functionality and individual Integrations emphasizes a plug-in–like architecture.
• Abstraction and Inheritance: The use of abstract classes (AbstractConnector, AbstractPipeline, etc.) allows for consistent extension and customization.
• Serverless Deployment: The design is meant for Apps Script “serverless” environments. No standalone server or cloud orchestration is required – the logic runs directly on the cloud environment provided by Google.
• Separation of Concerns: By separating configuration, connection logic, data pipeline, and storage, the design adheres to the principles of clear, modular design.
Relevant Technologies/Frameworks
• JavaScript and Google Apps Script – the primary programming environment.
• CLASP (Command Line Apps Script Projects) – indicated by the .clasp.json and .claspignore files in several integration folders.
• Google Sheets and possibly Google BigQuery – as target data storages for imported data.
• GitHub as the code hosting and collaboration tool, along with standard open-source documents (CONTRIBUTING.md, LICENSE, README.md).
Final Reminders for the Principal Software Engineer
• Be detailed – capture the major abstractions and how each integration ties back to the core.
• Tailor the level of detail to your audience – while some aspects can be overviewed, the core and integration patterns should be convincingly detailed.