How deepfence/threatmapper works
Main Components
• Management Console – This is the “face” of ThreatMapper. It includes the deepfence_frontend (with its UI components, reusable packages, dashboards etc.) and backend services (deepfence_server) that serve APIs and handle authentication, analytics, threat analysis, and integration with other components.
• Sensor Agents – Agents (deepfence_agent and deepfence_ctl), which run on endpoints (Kubernetes, Docker, bare metal, AWS Fargate, etc.) for runtime inspection and reporting.
• Cloud Scanner Tasks – These agent-less tasks periodically poll your cloud provider APIs (with Terraform-driven deployment modules) to gather configuration data and compliance metrics.
• Data Infrastructure & Supporting Systems – Data stores and supporting services such as:
• Relational database (deepfence_postgres) for structured data storage.
• Graph database (deepfence_neo4j) used for the ThreatGraph visualization.
• Caching and messaging (deepfence_redis and deepfence_kafka) for low-latency communication and system coordination.
• Worker components (deepfence_worker) for asynchronous tasks such as scans, report generation, and ingestion.
• Deployment and Integration – Numerous deployment scripts (helm charts, docker-compose files, Kubernetes manifests, etc.) and integration modules for external systems (email, Slack, Jira, etc.) that show how the product is packaged and rolled out.
• Cloud Scanner tasks (deployed via Terraform modules) that feed compliance and configuration data into the Management Console.
• Integration components for external services (for example, modules under pkg/integration and related directories) that communicate outwards (to Slack, email, etc.) for alerting and notifications.
• Finally, incorporate the deployment orchestration element, showing containerization (Dockerfiles, helm charts in deployment-scripts), CI/CD pipelines (GitHub Actions, CircleCI integrations) that manage releases and ensure a robust, microservices-oriented deployment.
Architectural Patterns and Design Principles
• The architecture follows a microservices, container-native approach with clearly defined separation of concerns: UI, API server, sensor data ingestion, and processing are decoupled.
• It embraces a distributed design which is visible in its use of separate “agents”, back-end workers, and external integrations.
• Communication between components is asynchronous (e.g., through Kafka/Redis) and uses well-defined API contracts.
• Strong observability is integrated, with real-time reporting, threat ranking, and a graphical representation (“ThreatGraph”) that benefits from a graph database backend.
Relevant Technologies and Frameworks
• Languages: Go (for backend server, worker, agent code), TypeScript/React (for frontend components).
• Containerization: Docker, Kubernetes (with helm charts in deployment-scripts).
• Infrastructure orchestration: Terraform modules for cloud scanning.
• Database: Postgres, Neo4j, Redis.
• Message Brokers: Kafka.
• CI/CD integrations: GitHub workflows, CircleCI, Jenkins, etc.