How phonepe/mantis works
Main Components
a. CLI & Input Layer
• This includes the primary entry points (launch.py, scheduler.py) as well as the command-line interface that accepts arguments (onboard or scan commands) and passes them into the system.
b. Configuration & Setup
• Look at the “configs” directory and the “mantis/config_parsers” modules. This layer is responsible for reading various configuration files (e.g., local, AWS, and other environment settings) and initializing logging or model configurations.
c. Modules / Core Scanning Engine
• The “mantis/modules” directory contains various specialized modules for different phases:
• Discovery Modules – Subdomains, certificates, etc.
• Recon Modules – Active host scanning (e.g., activehostscan), technology detection, WAF, etc.
• Scan Modules – Vulnerability scanning, misconfiguration checks, secrets scanning etc.
• Also, the “mantis/tool_base_classes” folder provides base classes for scanners that these modules extend from. This is important for showing how different tools or plugins integrate uniformly.
d. Workflow and Orchestration
• The “mantis/workflows” and “mantis/scan_orchestration” directories manage the sequence of scanning operations and control parallel execution (via thread pool and Ray for distributed scanning). This layer is key to coordinating the various steps of asset onboarding, recon, and scanning.
e. Persistence and Data Management
• Inside “mantis/db” you have the DB connection, CRUD operations (crud_assets, crud_vulnerabilities, etc.) and data models used to store, retrieve, and update scan results.
• Additionally, the dashboard support is facilitated by the inclusion of dashboard templates and related configurations (see “dashboard_templates”).
f. Utility & Helpers
• Directory “mantis/utils” includes essential shared functionality (argument parsing, configuration utilities, common request handling, notifications, etc.), which many parts of the system rely on.
• These help in maintaining consistency across modules.
g. External Integration and Deployment Infrastructure
• The project integrates external scanning tools (open source and custom scanners). Also, there’s support for AWS integrations (configs/aws_credentials, AWS Route53 support modules), as well as instructions around MongoDB and an AppSmith-based dashboard in the README.
• Docker-related files and setup scripts (inside the “setup” and “Dockerfile”) also indicate deployment as containerized microservices.
Relationships and Data Flow
• The CLI (launch.py & scheduler.py) is the starting point. It parses command-line options (see options in the README) and passes user input to the workflow layer.
• The workflow layer orchestrates the scanning process by invoking one or more modules (discovery, recon, scan) in a coordinated fashion.
• Each module (or tool) may call out into the DB layer to log results or fetch the necessary stored configurations.
• The scan orchestration layer (using threadpool_scan.py and ray_scan.py) manages parallel and distributed scanning, ensuring high throughput.
• Upon scan completion, results and logs are stored in a MongoDB instance. The dashboard then uses these entries for visualization (AppSmith dashboard integration).
• Configurations and settings are uniformly parsed at the start from the “configs” folder using “mantis/config_parsers.”
Architectural Patterns and Design Principles
• Modular/Plugin Architecture: Each scanning functionality is self-contained in its module. The base classes in “tool_base_classes” provide a uniform interface, encouraging extensibility (new modules can be added easily).
• Decoupled Workflow Orchestration: The workflow and orchestration layers (including distributed scanning with Ray) decouple the mechanism of triggering individual scans versus managing overall execution.
• Separation of Concerns: Different layers – from configuration parsing and DB operations to the scanning modules themselves – keep responsibilities separate.
• Containerization & Infrastructure as Code: Dockerfiles and setup scripts indicate that the deployment is designed to be repeatable and isolated via containers.
Detailed Component Breakdown
a. CLI Entry Points
• Launch.py, Scheduler.py: the initial input layer that triggers a scan or onboarding.
b. Configuration Layer
• Config Files (local, AWS)
• Config Parsers (config_client.py, config_models.py, logging_utils.py)
c. Workflow & Orchestration Layer
• Workflows (mantis_workflow.py, list_workflow.py, report_workflow.py)
• Scan Orchestration (ray_scan.py, threadpool_scan.py)
d. Scanning Modules
• Discovery Modules (e.g., Subfinder, SSLMate)
• Recon Modules (e.g., activehostscan/HTTPX, Wafw00f)
• Scan Modules (e.g., Nuclei, Corsy, Csper)
• Secrets Scanning (including submodules for secret scanning tools)
e. Core Abstractions
• Base Scanner Classes (apiScanner.py, baseScanner.py, toolScanner.py)
• Models for arguments and logs (args_model.py, tool_logs_model.py)
f. Data Layer
• Database Layer (database.py, CRUD operations, DB models)
g. Dashboard and External Tools
• Dashboard Templates (Mantis Dashboard.json)
• Instructions for Dockerized MongoDB and AppSmith