# google-marketing-solutions/crmint architecture

> Architecture diagram of the GitHub repository google-marketing-solutions/crmint, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-marketing-solutions/crmint
- Repository: https://github.com/google-marketing-solutions/crmint

## Overview

##### Project Type and Purpose
- CRMint is a full-stack, cloud-native data orchestration platform.
- It provides a web UI for non-engineers to define, schedule, and run data pipelines that move and process marketing data (BigQuery, GA, Ads, Vertex AI).
- It also includes a CLI deployment tool and Terraform modules for provisioning infrastructure on Google Cloud.

##### High-Level Components
a. Frontend
- Angular SPA served via NGINX (Dockerfile in frontend/)
- REST calls to the backend controller API
b. Backend Controller API
- Flask application (backend/controller/)
- Handles user requests: CRUD for pipelines, ML models, stages, jobs
- Enqueues tasks (via Cloud Pub/Sub or Celery tasks)
- Exposes health, metrics, and scheduling endpoints
c. Backend Jobs Workers
- Long-running workers (backend/jobs/) triggered by Pub/Sub or task queue
- Perform specific data operations: BigQuery queries, data exports/imports, GA updates, ML training/prediction on Vertex AI
d. Messaging / Task Queue
- Google Cloud Pub/Sub topics and subscriptions (defined in terraform/pubsub.tf)
- Optional Celery-ish task layer (`flask_tasks.py`)
e. Data Stores
- Cloud SQL (Postgres) for transactional metadata (pipelines, schedules, logs)
- BigQuery for data warehousing and ML
- Cloud Storage for intermediate/external files
f. Deployment & Infra
- Terraform modules (terraform/*.tf) for networking, IAM, IAP, databases, pub/sub, schedulers, services
- Cloud Build pipelines (cloudbuild.yaml in root & frontend/, backend/) to build Docker images & push to Container Registry
- CLI tool (cli/) to bootstrap and manage infrastructure and configuration

##### Interactions and Data Flows
- User ⇄ Frontend: HTTPS from user’s browser to Angular app
- Frontend → Controller API: REST (JSON) over HTTPS
- Controller → Database: ORM SQLAlchemy calls to Postgres
- Controller → Pub/Sub: publishes messages to trigger jobs
- Pub/Sub → Jobs Workers: subscribers consume messages
- Workers → BigQuery / GCS / GA / Ads / Vertex AI: SDK or REST calls
- Controller Scheduler: uses Cloud Scheduler (via terraform) to publish scheduled tasks

##### Architectural Patterns & Principles
- Microservice-like separation: Controller API vs. Jobs Workers
- Asynchronous messaging (Pub/Sub) for decoupling and scalability
- Infrastructure-as-Code (Terraform) for reproducibility
- CI/CD pipelines (GitHub Actions + Cloud Build) for automated builds & tests
- CLI wrapper for end-to-end provisioning

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        Browser["User Browser"]:::external
        Angular["Angular SPA"]:::compute
        NGINX["NGINX"]:::compute
        FE_CloudBuild["Frontend Cloud Build"]:::cicd
    end

    %% Application Layer
    subgraph "Application Layer"
        ControllerAPI["Flask Controller API"]:::compute
        BE_CloudBuild["Backend Cloud Build"]:::cicd
        DB["Cloud SQL (Postgres)"]:::database
    end

    %% Async Processing Layer
    subgraph "Async Processing Layer"
        PubSub["Pub/Sub"]:::cloud
        Workers["Jobs Workers"]:::compute
    end

    %% Data Plane
    subgraph "Data Plane"
        BigQuery["BigQuery"]:::cloud
        GCS["Cloud Storage"]:::cloud
    end

    %% External Services
    subgraph "External Services"
        GA["Google Analytics"]:::external
        Ads["Google Ads"]:::external
        VertexAI["Vertex AI"]:::external
    end

    %% Control Plane
    subgraph "Control Plane"
        Terraform["Terraform IaC"]:::infra
        Scheduler["Cloud Scheduler"]:::cloud
    end

    %% CI/CD & CLI
    subgraph "CI/CD & CLI"
        GitHubActions["GitHub Actions"]:::cicd
        Root_CloudBuild["Root Cloud Build"]:::cicd
        CLI["CLI Tool"]:::cicd
    end

    %% Connections
    Browser -->|"HTTPS"| Angular
    Angular -->|"Served by"| NGINX
    Angular -->|"REST/HTTPS"| ControllerAPI
    ControllerAPI -->|"ORM/SQLAlchemy"| DB
    ControllerAPI -->|"Publish Message"| PubSub
    PubSub -->|"Consume Message"| Workers
    Workers -->|"BigQuery SDK/REST"| BigQuery
    Workers -->|"GCS SDK/REST"| GCS
    Workers -->|"GA SDK/REST"| GA
    Workers -->|"Ads SDK/REST"| Ads
    Workers -->|"Vertex AI SDK/REST"| VertexAI
    Terraform --> Scheduler
    Terraform --> DB
    Terraform --> PubSub
    GitHubActions --> Root_CloudBuild
    Root_CloudBuild --> FE_CloudBuild
    Root_CloudBuild --> BE_CloudBuild
    CLI -.-> Terraform

    %% Click Events
    click Angular "https://github.com/google-marketing-solutions/crmint/tree/master/frontend/src/app/"
    click NGINX "https://github.com/google-marketing-solutions/crmint/blob/master/frontend/nginx.conf"
    click FE_CloudBuild "https://github.com/google-marketing-solutions/crmint/blob/master/frontend/cloudbuild.yaml"
    click ControllerAPI "https://github.com/google-marketing-solutions/crmint/tree/master/backend/controller/"
    click BE_CloudBuild "https://github.com/google-marketing-solutions/crmint/blob/master/backend/cloudbuild.yaml"
    click DB "https://github.com/google-marketing-solutions/crmint/blob/master/backend/controller/database.py"
    click PubSub "https://github.com/google-marketing-solutions/crmint/blob/master/terraform/pubsub.tf"
    click Workers "https://github.com/google-marketing-solutions/crmint/tree/master/backend/jobs/Dockerfile"
    click Terraform "https://github.com/google-marketing-solutions/crmint/blob/master/terraform/network.tf"
    click Scheduler "https://github.com/google-marketing-solutions/crmint/blob/master/terraform/schedulers.tf"
    click GitHubActions "https://github.com/google-marketing-solutions/crmint/tree/master/.github/workflows/"
    click Root_CloudBuild "https://github.com/google-marketing-solutions/crmint/blob/master/cloudbuild.yaml"
    click CLI "https://github.com/google-marketing-solutions/crmint/blob/master/cli/appcli.py"

    %% Styles
    classDef compute fill:#add8e6,stroke:#333,stroke-width:1px
    classDef database fill:#90ee90,stroke:#333,stroke-width:1px
    classDef cloud fill:#ffe4b5,stroke:#333,stroke-width:1px
    classDef infra fill:#f0e68c,stroke:#333,stroke-width:1px
    classDef external fill:#ffdead,stroke:#333,stroke-width:1px
    classDef cicd fill:#d3d3d3,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
