# owox/owox-data-marts architecture

> Architecture diagram of the GitHub repository owox/owox-data-marts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/owox/owox-data-marts
- Repository: https://github.com/owox/owox-data-marts

## Overview

##### Project type and purpose
- Monorepo housing a full-stack “Data Marts” platform and a library of JavaScript connectors.
- Purpose: enable data analysts to pull data from external marketing/financial APIs into Google Sheets or BigQuery, via a web UI, CLI, or Apps Script.

##### High-level components
A. Frontend (apps/web + packages/ui)
- Vite + React + TypeScript
- Uses a shared UI component library (`packages/ui`)
- Talks to the backend over REST endpoints (`apps/backend/src/data-marts/controllers…`)
B. Backend API (apps/backend)
- NestJS monolithic service
- Key modules: Data-Marts (controllers, services, use-cases), Scheduler, Authorization
- Persists user definitions & schedules in a relational database (SQLite by default)
- Exposes Swagger UI, global pipes, and static hosting of the built frontend
C. Connectors CLI (apps/owox + packages/connector-runner + packages/connectors)
- `apps/owox` – user-facing CLI for local development and BigQuery runs
- `packages/connector-runner` – framework to load & execute connector code in Node.js
- `packages/connectors` – library of individual connector implementations (FacebookAds, TikTokAds, etc.)
D. Data Storage Targets
- Google Sheets (Apps Script runtime)
- Google BigQuery (via `connector-runner`)
E. External Services
- 3rd-party APIs: Facebook Marketing, TikTok Ads, LinkedIn, Bank of Canada, etc.
- Google APIs: Sheets API, BigQuery API

##### Relationships and interactions
- Browser → Frontend (React UI)
- Frontend → Backend REST API (CRUD on Data-Mart definitions, schedules, triggers)
- Backend Scheduler → Triggers job notifications (direct or via Pub/Sub)
- Backend → Google Cloud (BigQuery) or Google Apps Script (Sheets)
- CLI (`owox`) → Connector-Runner → Connector implementations → External APIs → Writes to BigQuery
- Apps Script Templates → Connector implementations → External APIs → Writes to Google Sheets

##### Architectural patterns & principles
- Monorepo with npm workspaces: clear separation of concerns by folder (`apps/`, `packages/`)
- Modular NestJS (feature modules, common modules)
- Hexagonal/Ports & Adapters in connector-runner: core domain + CLI + infrastructure layers
- Reusable UI component library for consistency
- Scheduler pattern for periodic data pulls

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Actors
    Browser["Browser User"]:::actor
    CLIUser["CLI User"]:::actor

    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        Frontend["Web UI"]:::frontend
        CLIApp["OwoX CLI"]:::cli
    end

    %% Application Layer
    subgraph "Application Layer"
        direction TB
        Backend["Backend API (NestJS)"]:::backend
        Scheduler["Scheduler Module"]:::backend
        DataMartsModule["Data-Marts Module"]:::backend
        AuthContext["Authorization Context"]:::backend
        DB["Definitions & Schedules DB\n(SQLite)"]:::database
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        direction TB
        ConnectorRunner["Connector Runner\n(Node.js)"]:::runner
        Connectors["Connector Implementations"]:::connector
        AppsScriptRuntime["Apps Script Runtime"]:::infra
        GoogleSheets["Google Sheets API"]:::external
        BigQueryAPI["BigQuery API"]:::external
        ThirdPartyAPIs["3rd-Party APIs"]:::external
    end

    %% Relationships
    Browser -->|"Load UI\nHTTP (GET)"| Frontend
    CLIUser -->|"runConnector()"| CLIApp

    Frontend -->|"REST CRUD\nGET/POST /data-marts"| Backend
    Backend -->|"scheduleJob()"| Scheduler
    Scheduler -->|"storeJob()"| DB
    Scheduler -->|"triggerConnector()"| ConnectorRunner

    CLIApp -->|"uses"| ConnectorRunner
    ConnectorRunner -->|"load & exec"| Connectors
    Connectors -->|"HTTP requests"| ThirdPartyAPIs
    Connectors -->|"writeData()"| BigQueryAPI

    Backend -->|"Apps Script Execution"| AppsScriptRuntime
    AppsScriptRuntime -->|"invoke Connector"| Connectors
    Connectors -->|"HTTP requests"| ThirdPartyAPIs
    Connectors -->|"writeData()"| GoogleSheets

    %% Click Events
    click Frontend "https://github.com/owox/owox-data-marts/tree/main/apps/web"
    click Backend "https://github.com/owox/owox-data-marts/blob/main/apps/backend/src/main.ts"
    click DataMartsModule "https://github.com/owox/owox-data-marts/tree/main/apps/backend/src/data-marts"
    click Scheduler "https://github.com/owox/owox-data-marts/tree/main/apps/backend/src/common/scheduler"
    click AuthContext "https://github.com/owox/owox-data-marts/blob/main/apps/backend/src/common/authorization-context/authorization.context.ts"
    click CLIApp "https://github.com/owox/owox-data-marts/blob/main/apps/owox/src/index.ts"
    click ConnectorRunner "https://github.com/owox/owox-data-marts/tree/main/packages/connector-runner"
    click Connectors "https://github.com/owox/owox-data-marts/tree/main/packages/connectors"
    click GoogleSheets "https://github.com/owox/owox-data-marts/blob/main/packages/connectors/src/Storages/GoogleSheets/GoogleSheetsStorage.js"
    click BigQueryAPI "https://github.com/owox/owox-data-marts/blob/main/packages/connectors/src/Storages/GoogleBigQuery/GoogleBigQueryStorage.js"

    %% Styles
    classDef actor fill:#f9f,stroke:#333,stroke-width:1px
    classDef frontend fill:#b3d4fc,stroke:#036,stroke-width:1px
    classDef backend fill:#b3f3c8,stroke:#070,stroke-width:1px
    classDef cli fill:#e2c0ff,stroke:#702,stroke-width:1px
    classDef runner fill:#d0bfff,stroke:#702,stroke-width:1px
    classDef connector fill:#ffd1a9,stroke:#c60,stroke-width:1px
    classDef infra fill:#fff2b3,stroke:#a60,stroke-width:1px
    classDef external fill:#e0e0e0,stroke:#666,stroke-width:1px
    classDef database fill:#fff0f0,stroke:#900,stroke-width:1px
```
