How tracecathq/tracecat works
Project Type & Purpose
• Tracecat is a full-stack, open-source workflow automation platform for security and IT engineers.
• It consists of:
– A user-facing frontend (Next.js/React with a no-code UI)
– A Python-based backend REST API and orchestration engine
– A standalone “registry” CLI/tool for managing and publishing YAML-based integration templates
High-Level Architecture
• Clients (web browser or CLI) → Next.js Frontend & Registry CLI
• Frontend → HTTP(S) → Backend API (FastAPI/Starlette)
• Backend API → SQL (PostgreSQL), Temporal Server (workflow orchestration), Redis (caching/locks), optional S3 (secrets, artifacts)
• Temporal workers (Python processes) ← Backend schedules workflows/events → execute long-running tasks
• Registry CLI ← Python package tracecat_registry → interacts with local/remote template store (YAML files + schema validation)
Key Components to Model
A. Frontend (folder “frontend”)
– Next.js app
– Auth pages (OAuth, SAML), organization & workspace UIs, workflow builder, tables, registry browser
– API client layer (OpenAPI-generated TypeScript client)
B. API Server (folder “tracecat/api” + related modules)
– Routers (auth, cases, workflows, registry, secrets, webhooks, etc.)
– Services & business logic
– SQLAlchemy models + Alembic migrations
– Middleware (rate-limit, security, request tracing)
C. Orchestration & Workers
– tracecat/dsl scheduler & executor
– Temporal server & Python SDK
– Worker processes drive long-running tasks (email, HTTP calls, integrations)
D. Persistence & Infrastructure
– PostgreSQL (primary datastore)
– Redis (locking, caching, rate limits)
– Object store (S3-compatible—for secrets, artifacts)
– Temporal server (workflow orchestration)
E. Registry Package & CLI (folder “registry/tracecat_registry”)
– CLI entrypoint, template loader, schema validator, integrations library
– Publishes/validates YAML templates for integrations
Relationships & Data Flows
• User → (1) HTTP → Frontend → (2) REST calls → API Server → (3a) SQL (Postgres) / (3b) Temporal (schedule a workflow) / (3c) Redis → responds → Frontend → renders
• API Server → (4) Temporal Workers process workflow steps → call external services (AWS, Slack, LDAP, etc.) → update state back in Postgres or signal workflow progress
• Registry CLI → (A) Local filesystem (templates) or remote API → (B) Validates & publishes templates
Architectural Patterns & Principles
• Layered architecture (routers → services → data access)
• Event-driven orchestration (Temporal) for resilience & scaling
• Separation of concerns: UI vs. API vs. orchestration vs. CLI
• Infrastructure as code (Docker Compose, Terraform for AWS Fargate)
• CI/CD workflows (GitHub Actions for build, lint, test, release)
Detailing Enterprise vs. OSS
• “ee” directory as an extension module loaded by the backend for enterprise features (e.g., Kubernetes sandbox, advanced connectors)