How owox/owox-data-marts works
Project type and purpose
• Monorepo housing a full-stack “Data Marts” platform and a library of JavaScript connectors.
• Purpose: enable data analysts to pull data from external marketing/financial APIs into Google Sheets or BigQuery, via a web UI, CLI, or Apps Script.
High-level components
A. Frontend (apps/web + packages/ui)
• Vite + React + TypeScript
• Uses a shared UI component library (packages/ui)
• Talks to the backend over REST endpoints (apps/backend/src/data-marts/controllers…)
B. Backend API (apps/backend)
• NestJS monolithic service
• Key modules: Data-Marts (controllers, services, use-cases), Scheduler, Authorization
• Persists user definitions & schedules in a relational database (SQLite by default)
• Exposes Swagger UI, global pipes, and static hosting of the built frontend
C. Connectors CLI (apps/owox + packages/connector-runner + packages/connectors)
• apps/owox – user-facing CLI for local development and BigQuery runs
• packages/connector-runner – framework to load & execute connector code in Node.js
• packages/connectors – library of individual connector implementations (FacebookAds, TikTokAds, etc.)
D. Data Storage Targets
• Google Sheets (Apps Script runtime)
• Google BigQuery (via connector-runner)
E. External Services
• 3rd-party APIs: Facebook Marketing, TikTok Ads, LinkedIn, Bank of Canada, etc.
• Google APIs: Sheets API, BigQuery API
Relationships and interactions
• Browser → Frontend (React UI)
• Frontend → Backend REST API (CRUD on Data-Mart definitions, schedules, triggers)
• Backend Scheduler → Triggers job notifications (direct or via Pub/Sub)
• Backend → Google Cloud (BigQuery) or Google Apps Script (Sheets)
• CLI (owox) → Connector-Runner → Connector implementations → External APIs → Writes to BigQuery
• Apps Script Templates → Connector implementations → External APIs → Writes to Google Sheets
Architectural patterns & principles
• Monorepo with npm workspaces: clear separation of concerns by folder (apps/, packages/)
• Modular NestJS (feature modules, common modules)
• Hexagonal/Ports & Adapters in connector-runner: core domain + CLI + infrastructure layers
• Reusable UI component library for consistency
• Scheduler pattern for periodic data pulls