How google-marketing-solutions/crmint works
Project Type and Purpose
• CRMint is a full-stack, cloud-native data orchestration platform.
• It provides a web UI for non-engineers to define, schedule, and run data pipelines that move and process marketing data (BigQuery, GA, Ads, Vertex AI).
• It also includes a CLI deployment tool and Terraform modules for provisioning infrastructure on Google Cloud.
High-Level Components
a. Frontend
• Angular SPA served via NGINX (Dockerfile in frontend/)
• REST calls to the backend controller API
b. Backend Controller API
• Flask application (backend/controller/)
• Handles user requests: CRUD for pipelines, ML models, stages, jobs
• Enqueues tasks (via Cloud Pub/Sub or Celery tasks)
• Exposes health, metrics, and scheduling endpoints
c. Backend Jobs Workers
• Long-running workers (backend/jobs/) triggered by Pub/Sub or task queue
• Perform specific data operations: BigQuery queries, data exports/imports, GA updates, ML training/prediction on Vertex AI
d. Messaging / Task Queue
• Google Cloud Pub/Sub topics and subscriptions (defined in terraform/pubsub.tf)
• Optional Celery-ish task layer (flask_tasks.py)
e. Data Stores
• Cloud SQL (Postgres) for transactional metadata (pipelines, schedules, logs)
• BigQuery for data warehousing and ML
• Cloud Storage for intermediate/external files
f. Deployment & Infra
• Terraform modules (terraform/*.tf) for networking, IAM, IAP, databases, pub/sub, schedulers, services
• Cloud Build pipelines (cloudbuild.yaml in root & frontend/, backend/) to build Docker images & push to Container Registry
• CLI tool (cli/) to bootstrap and manage infrastructure and configuration
Interactions and Data Flows
• User ⇄ Frontend: HTTPS from user’s browser to Angular app
• Frontend → Controller API: REST (JSON) over HTTPS
• Controller → Database: ORM SQLAlchemy calls to Postgres
• Controller → Pub/Sub: publishes messages to trigger jobs
• Pub/Sub → Jobs Workers: subscribers consume messages
• Workers → BigQuery / GCS / GA / Ads / Vertex AI: SDK or REST calls
• Controller Scheduler: uses Cloud Scheduler (via terraform) to publish scheduled tasks
Architectural Patterns & Principles
• Microservice-like separation: Controller API vs. Jobs Workers
• Asynchronous messaging (Pub/Sub) for decoupling and scalability
• Infrastructure-as-Code (Terraform) for reproducibility
• CI/CD pipelines (GitHub Actions + Cloud Build) for automated builds & tests
• CLI wrapper for end-to-end provisioning