# deepfence/threatmapper architecture

> Architecture diagram of the GitHub repository deepfence/threatmapper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deepfence/threatmapper
- Repository: https://github.com/deepfence/threatmapper

## Overview

##### Main Components
- Management Console – This is the “face” of ThreatMapper. It includes the deepfence_frontend (with its UI components, reusable packages, dashboards etc.) and backend services (deepfence_server) that serve APIs and handle authentication, analytics, threat analysis, and integration with other components.
- Sensor Agents – Agents (deepfence_agent and deepfence_ctl), which run on endpoints (Kubernetes, Docker, bare metal, AWS Fargate, etc.) for runtime inspection and reporting.
- Cloud Scanner Tasks – These agent-less tasks periodically poll your cloud provider APIs (with Terraform-driven deployment modules) to gather configuration data and compliance metrics.
- Data Infrastructure & Supporting Systems – Data stores and supporting services such as:
- Relational database (deepfence_postgres) for structured data storage.
- Graph database (deepfence_neo4j) used for the ThreatGraph visualization.
- Caching and messaging (deepfence_redis and deepfence_kafka) for low-latency communication and system coordination.
- Worker components (deepfence_worker) for asynchronous tasks such as scans, report generation, and ingestion.
- Deployment and Integration – Numerous deployment scripts (helm charts, docker-compose files, Kubernetes manifests, etc.) and integration modules for external systems (email, Slack, Jira, etc.) that show how the product is packaged and rolled out.

- Cloud Scanner tasks (deployed via Terraform modules) that feed compliance and configuration data into the Management Console.
- Integration components for external services (for example, modules under pkg/integration and related directories) that communicate outwards (to Slack, email, etc.) for alerting and notifications.
- Finally, incorporate the deployment orchestration element, showing containerization (Dockerfiles, helm charts in deployment-scripts), CI/CD pipelines (GitHub Actions, CircleCI integrations) that manage releases and ensure a robust, microservices-oriented deployment.

##### Architectural Patterns and Design Principles
- The architecture follows a microservices, container-native approach with clearly defined separation of concerns: UI, API server, sensor data ingestion, and processing are decoupled.
- It embraces a distributed design which is visible in its use of separate “agents”, back-end workers, and external integrations.
- Communication between components is asynchronous (e.g., through Kafka/Redis) and uses well-defined API contracts.
- Strong observability is integrated, with real-time reporting, threat ranking, and a graphical representation (“ThreatGraph”) that benefits from a graph database backend.

##### Relevant Technologies and Frameworks
- Languages: Go (for backend server, worker, agent code), TypeScript/React (for frontend components).
- Containerization: Docker, Kubernetes (with helm charts in deployment-scripts).
- Infrastructure orchestration: Terraform modules for cloud scanning.
- Database: Postgres, Neo4j, Redis.
- Message Brokers: Kafka.
- CI/CD integrations: GitHub workflows, CircleCI, Jenkins, etc.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    subgraph "User Interaction"
        U["User (Admin)"]:::actor
    end

    %% Management Console (Frontend and Backend)
    subgraph "Management Console"
        FE["Management Console - Frontend UI"]:::frontend
        BE["Management Console - Backend API"]:::backend
        FE --> BE
    end

    U --> FE

    %% Sensor Agents
    subgraph "Sensor Agents"
        SA1["Sensor Agent - Runtime"]:::agent
        SA2["Sensor Agent - Control Tool"]:::agent
    end

    SA1 -->|"telemetry"| BE
    SA2 -->|"control"| BE

    %% Cloud Scanner Tasks
    SC["Cloud Scanner Tasks"]:::scanner
    SC -->|"configData"| BE

    %% Data Infrastructure & Supporting Systems
    subgraph "Data Infrastructure"
        DB1["Postgres Database"]:::infra
        DB2["Neo4j Graph Database"]:::infra
        Cache["Redis Caching/Messaging"]:::infra
        Msg["Kafka Messaging"]:::infra
        Worker["Worker Processes"]:::infra
    end

    BE -->|"reads/writes"| DB1
    BE -->|"reads/writes"| DB2
    BE -->|"pub/sub"| Cache
    BE -->|"pub/sub"| Msg
    BE -->|"triggers"| Worker

    %% External Integrations
    subgraph "External Integrations"
        Ext["External Integrations"]:::integration
    end

    BE -->|"alerts"| Ext

    %% Deployment and Orchestration
    subgraph "Deployment & Orchestration"
        Dep["Deployment Scripts"]:::deployment
        CI["CI/CD Integrations"]:::deployment
    end

    Dep --> FE
    Dep --> BE
    CI --> FE
    CI --> BE
    CI --> SA1
    CI --> SA2

    %% Click Events
    click FE "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_frontend"
    click BE "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_server"
    click SA1 "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_agent"
    click SA2 "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_ctl"
    click SC "https://github.com/deepfence/threatmapper/blob/release-2.5/deepfence_bootstrapper/router/cloud_scanner.go"
    click DB1 "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_postgres"
    click DB2 "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_neo4j"
    click Cache "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_redis"
    click Msg "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_kafka"
    click Worker "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_worker"
    click Ext "https://github.com/deepfence/threatmapper/tree/release-2.5/deepfence_server/pkg/integration"
    click Dep "https://github.com/deepfence/threatmapper/tree/release-2.5/deployment-scripts"
    click CI "https://github.com/deepfence/threatmapper/tree/release-2.5/ci-cd-integrations"

    %% Styles
    classDef frontend fill:#f9d6c1,stroke:#333,stroke-width:2px;
    classDef backend fill:#c1d6f9,stroke:#333,stroke-width:2px;
    classDef agent fill:#d6f9c1,stroke:#333,stroke-width:2px;
    classDef scanner fill:#f9c1d6,stroke:#333,stroke-width:2px;
    classDef infra fill:#f3f3a3,stroke:#333,stroke-width:2px;
    classDef integration fill:#c1f9f1,stroke:#333,stroke-width:2px;
    classDef deployment fill:#f1c1f9,stroke:#333,stroke-width:2px;
    classDef actor fill:#ffe6a7,stroke:#333,stroke-width:2px;
```
