# phonepe/mantis architecture

> Architecture diagram of the GitHub repository phonepe/mantis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phonepe/mantis
- Repository: https://github.com/phonepe/mantis

## Overview

##### Main Components

a. CLI & Input Layer
- This includes the primary entry points (launch.py, scheduler.py) as well as the command-line interface that accepts arguments (onboard or scan commands) and passes them into the system.

b. Configuration & Setup
- Look at the “configs” directory and the “mantis/config_parsers” modules. This layer is responsible for reading various configuration files (e.g., local, AWS, and other environment settings) and initializing logging or model configurations.

c. Modules / Core Scanning Engine
- The “mantis/modules” directory contains various specialized modules for different phases:
- Discovery Modules – Subdomains, certificates, etc.
- Recon Modules – Active host scanning (e.g., activehostscan), technology detection, WAF, etc.
- Scan Modules – Vulnerability scanning, misconfiguration checks, secrets scanning etc.
- Also, the “mantis/tool_base_classes” folder provides base classes for scanners that these modules extend from. This is important for showing how different tools or plugins integrate uniformly.

d. Workflow and Orchestration
- The “mantis/workflows” and “mantis/scan_orchestration” directories manage the sequence of scanning operations and control parallel execution (via thread pool and Ray for distributed scanning). This layer is key to coordinating the various steps of asset onboarding, recon, and scanning.

e. Persistence and Data Management
- Inside “mantis/db” you have the DB connection, CRUD operations (crud_assets, crud_vulnerabilities, etc.) and data models used to store, retrieve, and update scan results.
- Additionally, the dashboard support is facilitated by the inclusion of dashboard templates and related configurations (see “dashboard_templates”).

f. Utility & Helpers
- Directory “mantis/utils” includes essential shared functionality (argument parsing, configuration utilities, common request handling, notifications, etc.), which many parts of the system rely on.
- These help in maintaining consistency across modules.

g. External Integration and Deployment Infrastructure
- The project integrates external scanning tools (open source and custom scanners). Also, there’s support for AWS integrations (configs/aws_credentials, AWS Route53 support modules), as well as instructions around MongoDB and an AppSmith-based dashboard in the README.
- Docker-related files and setup scripts (inside the “setup” and “Dockerfile”) also indicate deployment as containerized microservices.

##### Relationships and Data Flow

- The CLI (launch.py & scheduler.py) is the starting point. It parses command-line options (see options in the README) and passes user input to the workflow layer.
- The workflow layer orchestrates the scanning process by invoking one or more modules (discovery, recon, scan) in a coordinated fashion.
- Each module (or tool) may call out into the DB layer to log results or fetch the necessary stored configurations.
- The scan orchestration layer (using threadpool_scan.py and ray_scan.py) manages parallel and distributed scanning, ensuring high throughput.
- Upon scan completion, results and logs are stored in a MongoDB instance. The dashboard then uses these entries for visualization (AppSmith dashboard integration).
- Configurations and settings are uniformly parsed at the start from the “configs” folder using “mantis/config_parsers.”

##### Architectural Patterns and Design Principles

- Modular/Plugin Architecture: Each scanning functionality is self-contained in its module. The base classes in “tool_base_classes” provide a uniform interface, encouraging extensibility (new modules can be added easily).
- Decoupled Workflow Orchestration: The workflow and orchestration layers (including distributed scanning with Ray) decouple the mechanism of triggering individual scans versus managing overall execution.
- Separation of Concerns: Different layers – from configuration parsing and DB operations to the scanning modules themselves – keep responsibilities separate.
- Containerization & Infrastructure as Code: Dockerfiles and setup scripts indicate that the deployment is designed to be repeatable and isolated via containers.

##### Detailed Component Breakdown

a. CLI Entry Points
- Launch.py, Scheduler.py: the initial input layer that triggers a scan or onboarding.

b. Configuration Layer
- Config Files (local, AWS)
- Config Parsers (config_client.py, config_models.py, logging_utils.py)

c. Workflow & Orchestration Layer
- Workflows (mantis_workflow.py, list_workflow.py, report_workflow.py)
- Scan Orchestration (ray_scan.py, threadpool_scan.py)

d. Scanning Modules
- Discovery Modules (e.g., Subfinder, SSLMate)
- Recon Modules (e.g., activehostscan/HTTPX, Wafw00f)
- Scan Modules (e.g., Nuclei, Corsy, Csper)
- Secrets Scanning (including submodules for secret scanning tools)

e. Core Abstractions
- Base Scanner Classes (apiScanner.py, baseScanner.py, toolScanner.py)
- Models for arguments and logs (args_model.py, tool_logs_model.py)

f. Data Layer
- Database Layer (database.py, CRUD operations, DB models)

g. Dashboard and External Tools
- Dashboard Templates (Mantis Dashboard.json)
- Instructions for Dockerized MongoDB and AppSmith

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Input Layer
    subgraph "CLI & Input Layer"
        CLI1["launch.py"]:::cli
        CLI2["scheduler.py"]:::cli
    end

    %% Configuration & Setup
    subgraph "Configuration & Setup"
        subgraph "Config Files"
            CF1["aws_config"]:::configFile
            CF2["aws_credentials"]:::configFile
            CF3["local.yml"]:::configFile
        end
        subgraph "Config Parsers"
            CP1["config_client.py"]:::config
            CP2["config_models.py"]:::config
            CP3["logging_utils.py"]:::config
        end
    end

    %% Workflow & Orchestration
    subgraph "Workflow & Orchestration"
        subgraph "Workflows"
            WF1["mantis_workflow.py"]:::workflow
            WF2["list_workflow.py"]:::workflow
            WF3["report_workflow.py"]:::workflow
        end
        subgraph "Scan Orchestration"
            SO1["ray_scan.py"]:::workflow
            SO2["threadpool_scan.py"]:::workflow
        end
    end

    %% Modules / Core Scanning Engine
    subgraph "Modules / Core Scanning Engine"
        subgraph "Scanning Modules"
            SM1["discovery"]:::modules
            SM2["activerecon"]:::modules
            SM3["activehostscan"]:::modules
            SM4["dns"]:::modules
            SM5["prerecon"]:::modules
            SM6["scan"]:::modules
            SM7["secretscanner"]:::modules
        end
        subgraph "Tool Base Classes"
            TB1["apiScanner.py"]:::modules
            TB2["baseScanner.py"]:::modules
            TB3["toolScanner.py"]:::modules
        end
    end

    %% Persistence and Data Management
    subgraph "Persistence and Data Management"
        DB1["database.py"]:::db
        DB2["crud_assets.py"]:::db
        DB3["crud_extended_assets.py"]:::db
        DB4["crud_vulnerabilities.py"]:::db
        DB5["db_models.py"]:::db
    end

    %% Utility & Helpers
    subgraph "Utility & Helpers"
        U1["args_parse.py"]:::utils
        U2["asset_type.py"]:::utils
        U3["base_request.py"]:::utils
        U4["common_utils.py"]:::utils
        U5["config_utils.py"]:::utils
        U6["crud_utils.py"]:::utils
        U7["list_assets.py"]:::utils
        U8["list_subcommand_utils.py"]:::utils
        U9["notifications.py"]:::utils
        U10["tool_utils.py"]:::utils
    end

    %% Dashboard and External Integration
    subgraph "Dashboard and External Integration"
        subgraph "Dashboard"
            D1["Mantis Dashboard.json"]:::dashboard
        end
        subgraph "Deployment"
            DEP1["Dockerfile"]:::dashboard
            DEP2["setup/docker"]:::dashboard
        end
    end

    %% Data Flow Connections
    %% CLI to Configuration
    CLI1 -->|"init_config"| CP1
    CLI2 -->|"init_config"| CP1
    CLI1 -->|"fetch_config"| CF1
    CLI2 -->|"fetch_config"| CF1

    %% Configuration to Workflow & Orchestration
    CP1 -->|"provides_config"| WF1
    CF1 -->|"loads_settings"| WF1

    %% Workflow & Orchestration to Modules / Core Scanning Engine
    WF1 -->|"orchestrates"| SM1
    WF1 -->|"orchestrates"| SM3
    SO2 -->|"parallel_scan"| SM6

    %% Modules to Persistence and Data Management
    SM1 -->|"logs_results"| DB1
    SM3 -->|"records_info"| DB2
    SM4 -->|"saves_dns"| DB3
    SM7 -->|"stores_vulns"| DB4
    TB1 -->|"base_func"| DB5

    %% Persistence to Dashboard
    DB1 -->|"scan_data"| D1
    DB2 -->|"asset_data"| D1

    %% Deployment to Dashboard
    DEP1 -->|"deploys"| D1
    DEP2 -->|"deploys"| D1

    %% Utility & Helpers support dotted connections
    U1 -.-> WF1
    U4 -.-> SM1
    U5 -.-> CP1
    U10 -.-> SM6

    %% Styling Classes
    classDef cli fill:#F7C6C7,stroke:#8B0000,stroke-width:2px;
    classDef config fill:#C6E2FF,stroke:#0B3D91,stroke-width:2px;
    classDef configFile fill:#D0E9F5,stroke:#0B3D91,stroke-width:2px;
    classDef workflow fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px;
    classDef modules fill:#FFF9C4,stroke:#F9A825,stroke-width:2px;
    classDef db fill:#FFE0B2,stroke:#E65100,stroke-width:2px;
    classDef utils fill:#D1C4E9,stroke:#4A148C,stroke-width:2px;
    classDef dashboard fill:#B2EBF2,stroke:#006064,stroke-width:2px;

    %% Click Events
    click CLI1 "https://github.com/phonepe/mantis/blob/main/launch.py"
    click CLI2 "https://github.com/phonepe/mantis/blob/main/scheduler.py"
    click CF1 "https://github.com/phonepe/mantis/tree/main/configs/aws_config"
    click CF2 "https://github.com/phonepe/mantis/tree/main/configs/aws_credentials"
    click CF3 "https://github.com/phonepe/mantis/blob/main/configs/local.yml"
    click CP1 "https://github.com/phonepe/mantis/blob/main/mantis/config_parsers/config_client.py"
    click CP2 "https://github.com/phonepe/mantis/blob/main/mantis/config_parsers/config_models.py"
    click CP3 "https://github.com/phonepe/mantis/blob/main/mantis/config_parsers/logging_utils.py"
    click WF1 "https://github.com/phonepe/mantis/blob/main/mantis/workflows/mantis_workflow.py"
    click WF2 "https://github.com/phonepe/mantis/blob/main/mantis/workflows/list_workflow.py"
    click WF3 "https://github.com/phonepe/mantis/blob/main/mantis/workflows/report_workflow.py"
    click SO1 "https://github.com/phonepe/mantis/blob/main/mantis/scan_orchestration/ray_scan.py"
    click SO2 "https://github.com/phonepe/mantis/blob/main/mantis/scan_orchestration/threadpool_scan.py"
    click SM1 "https://github.com/phonepe/mantis/tree/main/mantis/modules/discovery"
    click SM2 "https://github.com/phonepe/mantis/tree/main/mantis/modules/activerecon"
    click SM3 "https://github.com/phonepe/mantis/tree/main/mantis/modules/activehostscan"
    click SM4 "https://github.com/phonepe/mantis/tree/main/mantis/modules/dns"
    click SM5 "https://github.com/phonepe/mantis/tree/main/mantis/modules/prerecon"
    click SM6 "https://github.com/phonepe/mantis/tree/main/mantis/modules/scan"
    click SM7 "https://github.com/phonepe/mantis/tree/main/mantis/modules/secretscanner"
    click TB1 "https://github.com/phonepe/mantis/blob/main/mantis/tool_base_classes/apiScanner.py"
    click TB2 "https://github.com/phonepe/mantis/blob/main/mantis/tool_base_classes/baseScanner.py"
    click TB3 "https://github.com/phonepe/mantis/blob/main/mantis/tool_base_classes/toolScanner.py"
    click DB1 "https://github.com/phonepe/mantis/blob/main/mantis/db/database.py"
    click DB2 "https://github.com/phonepe/mantis/blob/main/mantis/db/crud_assets.py"
    click DB3 "https://github.com/phonepe/mantis/blob/main/mantis/db/crud_extended_assets.py"
    click DB4 "https://github.com/phonepe/mantis/blob/main/mantis/db/crud_vulnerabilities.py"
    click DB5 "https://github.com/phonepe/mantis/blob/main/mantis/db/db_models.py"
    click U1 "https://github.com/phonepe/mantis/blob/main/mantis/utils/args_parse.py"
    click U2 "https://github.com/phonepe/mantis/blob/main/mantis/utils/asset_type.py"
    click U3 "https://github.com/phonepe/mantis/blob/main/mantis/utils/base_request.py"
    click U4 "https://github.com/phonepe/mantis/blob/main/mantis/utils/common_utils.py"
    click U5 "https://github.com/phonepe/mantis/blob/main/mantis/utils/config_utils.py"
    click U6 "https://github.com/phonepe/mantis/blob/main/mantis/utils/crud_utils.py"
    click U7 "https://github.com/phonepe/mantis/blob/main/mantis/utils/list_assets.py"
    click U8 "https://github.com/phonepe/mantis/blob/main/mantis/utils/list_subcommand_utils.py"
    click U9 "https://github.com/phonepe/mantis/blob/main/mantis/utils/notifications.py"
    click U10 "https://github.com/phonepe/mantis/blob/main/mantis/utils/tool_utils.py"
    click D1 "https://github.com/phonepe/mantis/blob/main/dashboard_templates/Mantis Dashboard.json"
    click DEP1 "https://github.com/phonepe/mantis/tree/main/Dockerfile"
    click DEP2 "https://github.com/phonepe/mantis/tree/main/setup/docker"
```
