# vemetric/vemetric architecture

> Architecture diagram of the GitHub repository vemetric/vemetric, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vemetric/vemetric
- Repository: https://github.com/vemetric/vemetric

## Overview

##### Project Type and Purpose
- Full-stack, open-source Web & Product Analytics platform
- Real-time event ingestion, processing pipelines, dashboard UI, billing & user management

##### Top-Level Components
A. Frontend (apps/webapp)
– React + Vite app
– Pages & components for dashboard, funnels, events, user detail, settings, onboarding
– trpc client for backend API calls
– Snippet that posts tracking events to the “Hub”

B. Backend API (apps/backend)
– Node.js/TypeScript + tRPC server
– REST/trpc routes: billing, dashboard queries, events, filters, funnels, users, organization, Paddle webhooks
– Uses Prisma client (packages/database) and ClickHouse client (packages/clickhouse)
– Handles authentication (lucia), authorization, rate limiting, logging

C. Ingestion Hub (apps/hub)
– Node.js ingestion service
– Serves the client snippet and receives raw events
– Enriches events (cookies, bots, sessions)
– Pushes jobs/messages into Redis queues

D. Worker Pool (apps/worker)
– Background workers (Bull) subscribed to queues: event-worker, session-worker, user-worker, device-worker, email-worker, etc.
– Processes queues to:
- Write to relational DB (Prisma/Postgres via packages/database)
- Write to analytics DB (ClickHouse via packages/clickhouse)
- Trigger email drip sequences (packages/email + Postmark)
- Send notifications or update usage metrics

E. Queue Infrastructure (packages/queues + apps/bullboard)
– Redis as message broker + Bull for job queues
– apps/bullboard provides a UI to monitor queue statuses

F. Supporting Utilities
– packages/common: shared data types, utility functions, filters
– packages/logger: structured logging across services
– apps/dev-proxy: local dev proxy routing traffic to each app
– apps/health-check: periodic health-check pings
– Docker & docker-compose orchestrate containers: backend, hub, worker, Redis, ClickHouse, Postgres

G. Data Stores
– Postgres (Prisma) for core relational data: users, projects, billing, feature flags
– ClickHouse for analytical time-series & event data
– Redis for queues and session caching

H. External Services
– Paddle (billing, webhooks)
– Postmark (transactional & drip emails)
– Crisp Chat (embedded support chat)

##### Interactions & Data Flow
1. User visits site → webapp (client) loads tracking snippet from Hub
2. Client records pageviews, events → sends POST to Hub API
3. Hub enriches event → pushes job to Redis queue
4. Worker picks up job → processes event:
- Writes raw analytics to ClickHouse
- Updates session/user info in Postgres
- Emits further jobs (e.g., email drip)
5. Frontend trpc calls → Backend API → reads from Postgres or ClickHouse → returns dashboard/funnel/filter data
6. Billing events handled by backend via Paddle integration → updates billing-info in Postgres
7. Email-worker uses packages/email to send transactional or sequence emails

##### Architectural Patterns & Principles
– Microservices-style apps under a single monorepo (pnpm + Turborepo)
– Event-driven pipeline: Hub → Redis queues → Workers
– CQRS-like separation:
- Command path: event ingestion & processing
- Query path: ClickHouse + API for dashboards
– Modular packages for cross-cutting concerns (common, queues, logger, clickhouse, database, email)
– Containerized services for isolation & easy deployment

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User and Frontend
    User["User Browser"]:::user
    Webapp["Webapp (React/Vite)"]:::frontend
    User -->|"visits site"| Webapp
    Webapp -->|"loads snippet from Hub"| Hub

    %% Services Layer
    subgraph "Core Services"
        direction TB
        Hub["Ingestion Hub\n(Node.js/TypeScript)"]:::service
        Backend["Backend API\n(tRPC/REST, Node.js/TypeScript)"]:::service
    end
    Webapp -->|"trackEvent POST"| Hub
    Webapp -->|"trpc calls"| Backend

    %% Queues and Monitoring
    subgraph "Queue Infrastructure"
        direction TB
        Redis["Redis (Broker)"]:::queue
        Queues["Queue Definitions\n(packages/queues)"]:::queue
        BullBoard["BullBoard UI"]:::frontend
    end
    Hub -->|"enqueue job"| Redis
    Redis -->|"jobs via Queues"| Queues
    Queues -->|"monitor"| BullBoard

    %% Worker Pool
    WorkerPool["Worker Pool\n(Bull Workers)"]:::service
    Redis -->|"dequeue jobs"| WorkerPool

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        Postgres["Postgres (Prisma)"]:::datastore
        ClickHouse["ClickHouse"]:::datastore
    end
    WorkerPool -->|"writes core data"| Postgres
    WorkerPool -->|"writes analytics"| ClickHouse
    Backend -->|"reads/writes core"| Postgres
    Backend -->|"reads analytics"| ClickHouse

    %% Supporting Utilities
    subgraph "Shared Utilities"
        direction TB
        Common["Common Utils"]:::service
        Logger["Structured Logger"]:::service
        DBClient["Prisma Client"]:::service
        CHClient["ClickHouse Client"]:::service
        EmailPkg["Email Service\n(packages/email)"]:::service
    end
    Hub --> Common
    Hub --> Logger
    Backend --> Common
    Backend --> Logger
    Backend --> DBClient
    Backend --> CHClient
    WorkerPool --> Common
    WorkerPool --> Logger
    WorkerPool --> DBClient
    WorkerPool --> CHClient
    WorkerPool --> EmailPkg

    %% Dev and Health
    DevProxy["Dev Proxy"]:::service
    Health["Health Check"]:::service
    DevProxy --> Backend
    DevProxy --> Hub
    DevProxy --> WorkerPool
    Health --> Backend
    Health --> Hub
    Health --> WorkerPool

    %% External Services
    Paddle["Paddle (Billing)"]:::external
    Postmark["Postmark (Email)"]:::external
    Crisp["Crisp Chat"]:::external
    Backend -->|billing webhooks| Paddle
    WorkerPool -->|send emails| Postmark
    Webapp -->|embedded chat| Crisp

    %% Click Events
    click Webapp "https://github.com/vemetric/vemetric/tree/main/apps/webapp"
    click Backend "https://github.com/vemetric/vemetric/tree/main/apps/backend"
    click Hub "https://github.com/vemetric/vemetric/tree/main/apps/hub"
    click WorkerPool "https://github.com/vemetric/vemetric/tree/main/apps/worker"
    click Queues "https://github.com/vemetric/vemetric/tree/main/packages/queues"
    click BullBoard "https://github.com/vemetric/vemetric/tree/main/apps/bullboard"
    click Common "https://github.com/vemetric/vemetric/tree/main/packages/common"
    click Logger "https://github.com/vemetric/vemetric/tree/main/packages/logger"
    click DBClient "https://github.com/vemetric/vemetric/tree/main/packages/database"
    click CHClient "https://github.com/vemetric/vemetric/tree/main/packages/clickhouse"
    click EmailPkg "https://github.com/vemetric/vemetric/tree/main/packages/email"
    click DevProxy "https://github.com/vemetric/vemetric/tree/main/apps/dev-proxy"
    click Health "https://github.com/vemetric/vemetric/tree/main/apps/health-check"

    %% Styles
    classDef user fill:#BBDEFB,stroke:#1E88E5;
    classDef frontend fill:#90CAF9,stroke:#1565C0;
    classDef service fill:#A5D6A7,stroke:#2E7D32;
    classDef queue fill:#FFE082,stroke:#EF6C00;
    classDef datastore fill:#FFAB91,stroke:#E64A19;
    classDef external fill:#E1BEE7,stroke:#6A1B9A;
```
