# gh05tcrew/pentestagent architecture

> Architecture diagram of the GitHub repository gh05tcrew/pentestagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gh05tcrew/pentestagent
- Repository: https://github.com/gh05tcrew/pentestagent

## Overview

##### Project Type and Purpose
- GHOSTCREW is a command-line AI “red team” assistant (open-source tool) built in Python.
- It uses Large Language Models (via OpenAI API), a Retrieval-Augmented Generation (RAG) knowledge base, and MCP protocol to call external pentesting tools.
- Its main functions are natural-language driven pen-testing workflows, autonomous agent mode, interactive tool configuration, and markdown report generation.

##### File Structure
Top-level directories and roles:
- config/: application and constant definitions (app_config.py, constants.py).
- core/: the heart of the assistant:
- agent_mode_controller.py, agent_runner.py – orchestrate autonomous agents with task trees
- model_manager.py – wraps OpenAI API calls, streaming, model parameters
- pentest_agent.py, ptt_reasoning.py – decision-making logic
- task_tree_manager.py – builds/manages Pentesting Task Trees (PTT)
- rag/: RAG subsystem
- knowledge_base.py – loads text files, vector store abstraction
- embedding.py – handles embedding calls to LLM API
- tools/: MCP tooling integration
- configure_mcp.py, mcp_manager.py – read/write mcp.json, launch/communicate with MCP servers
- workflows/: predefined, repeatable pentesting sequences
- workflow_definitions.py, workflow_engine.py
- reporting/: markdown report generation (generators.py)
- ui/: CLI interface
- menu_system.py – top-level menus (chat vs workflow vs agent)
- conversation_manager.py – handles user I/O, multi-turn contexts
- knowledge/: optional RAG content (wordlist.txt, plus user-supplied files)
- main.py: application entrypoint loading config, kicking off UI

##### Key README Insights
- Modes: Chat, Workflows, Agent
- MCP server integrations (Node.js or Python backends) configured via mcp.json
- Knowledge-base enhancement optional, triggered at startup
- Streaming responses, conversation history, multi-line input

1. CLI UI Layer
– menu_system.py
– conversation_manager.py
2. Configuration Layer
– config/app_config.py, constants.py
– mcp.json (persistence)
3. Core Logic Layer
– Agent Mode Controller & Runner
– Task Tree Manager
– Pentest Agent & Reasoning
– Model Manager (LLM interface)
4. RAG Subsystem (optional path)
– Knowledge Base Loader
– Embedding Service
– Local vector store (in-memory or disk)
5. Tools Integration Layer
– MCP Manager
– Configure MCP Tool script
– External MCP Servers (node- or python-based tool daemons)
6. Workflows Engine
– Workflow Definitions
– Workflow Engine
7. Reporting Layer
– Markdown Report Generator
8. External Services
– OpenAI API (LLM & Embeddings)
– MCP Tool Servers (nmap, metasploit, ffuf, etc.)

- UI → Core Logic (user commands mapped to controllers)
- Core Logic → Model Manager → OpenAI API (prompts, responses)
- Core Logic (when RAG enabled) → RAG Subsystem → Model Manager → OpenAI API
- Core Logic/Workflows → MCP Manager → External MCP Servers → Results → Core Logic
- Core Logic/Workflows → Reporting → Markdown File Output
- Persistent Config (mcp.json, config files) read at startup & updated by Tools layer

C. Architectural Patterns & Principles
– Layered architecture (UI / Application / Integration / External)
– Plugin/Adapter pattern for MCP tools (mcp_manager abstracts heterogeneous tool servers)
– Strategy pattern in workflows (each workflow definition encapsulates a sequence of tool calls)
– Pipeline for RAG (load corpus → embed → vector query → prompt enrichment)
– Agent-based control for autonomous pentesting (task trees + reasoning)

D. Key Technologies & Frameworks
– Python 3, standard libraries
– OpenAI API (chat completions + embeddings)
– Node.js or Python daemons for MCP tools (via npx or local servers)
– JSON config (mcp.json)
– CLI I/O handling (built-in input/print, optional streaming)

##### Detail, Not Overthinking
– Break the system into as many logical components as above.
– Capture conditional paths (RAG vs pure LLM chat, agent vs workflow vs chat).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        appConfig["app_config.py"]:::config
        constants["constants.py"]:::config
        mcpjson["mcp.json"]:::configFile
    end

    %% CLI UI Layer
    subgraph "CLI UI Layer"
        direction TB
        menuSystem["menu_system.py"]:::ui
        conversationManager["conversation_manager.py"]:::ui
    end

    %% Core Logic Layer
    subgraph "Core Logic Layer"
        direction TB
        agentMode["agent_mode_controller.py"]:::core
        agentRunner["agent_runner.py"]:::core
        taskTree["task_tree_manager.py"]:::core
        pentestAgent["pentest_agent.py"]:::core
        pttReasoning["ptt_reasoning.py"]:::core
        modelManager["model_manager.py"]:::core
    end

    %% RAG Subsystem
    subgraph "RAG Subsystem"
        direction TB
        knowledgeBase["knowledge_base.py"]:::rag
        embedding["embedding.py"]:::rag
        vectorStore["Local Vector Store"]:::storage
    end

    %% Tools Integration Layer
    subgraph "Tools Integration Layer"
        direction TB
        configureMCP["configure_mcp.py"]:::tools
        mcpManager["mcp_manager.py"]:::tools
    end

    %% Workflows Engine
    subgraph "Workflows Engine"
        direction TB
        workflowDefs["workflow_definitions.py"]:::workflow
        workflowEngine["workflow_engine.py"]:::workflow
    end

    %% Reporting Layer
    subgraph "Reporting Layer"
        direction TB
        reportGen["generators.py"]:::report
        reportFile["Markdown Report (.md)"]:::storage
    end

    %% Entry Point
    main["main.py"]:::entry

    %% External Services
    openai["OpenAI API"]:::external
    mcpServers["MCP Tool Servers"]:::external
       
    %% Startup Flow
    main -->|load config| appConfig
    main --> constants
    main --> mcpjson
    main -->|optional load| knowledgeBase
    main -->|init tools| configureMCP

    %% UI to Core
    menuSystem -->|user command| agentMode
    menuSystem --> conversationManager
    conversationManager -->|chat input| modelManager

    %% Core Logic Flows
    agentMode --> agentRunner
    agentRunner --> taskTree
    taskTree --> pttReasoning
    pttReasoning --> pentestAgent
    pentestAgent --> modelManager

    %% Model Manager to OpenAI
    modelManager -->|LLM & embeddings| openai

    %% RAG Path
    knowledgeBase -->|load texts| vectorStore
    embedding -->|embed query| openai
    modelManager -->|augmented prompt| embedding
    modelManager -->|retrieve vectors| vectorStore

    %% Tools Integration
    agentMode --> mcpManager
    workflowEngine --> mcpManager
    mcpManager -->|run tool: nmap, metasploit, etc.| mcpServers
    mcpServers -->|results| mcpManager
    mcpManager --> pentestAgent

    %% Workflows Flow
    menuSystem -->|select workflow| workflowEngine
    workflowEngine --> workflowDefs
    workflowEngine --> mcpManager
    workflowEngine --> reportGen

    %% Reporting Flow
    pentestAgent --> reportGen
    reportGen --> reportFile

    %% Click Events
    click appConfig "https://github.com/gh05tcrew/pentestagent/blob/main/config/app_config.py"
    click constants "https://github.com/gh05tcrew/pentestagent/blob/main/config/constants.py"
    click mcpjson "https://github.com/gh05tcrew/pentestagent/blob/main/mcp.json"
    click menuSystem "https://github.com/gh05tcrew/pentestagent/blob/main/ui/menu_system.py"
    click conversationManager "https://github.com/gh05tcrew/pentestagent/blob/main/ui/conversation_manager.py"
    click agentMode "https://github.com/gh05tcrew/pentestagent/blob/main/core/agent_mode_controller.py"
    click agentRunner "https://github.com/gh05tcrew/pentestagent/blob/main/core/agent_runner.py"
    click taskTree "https://github.com/gh05tcrew/pentestagent/blob/main/core/task_tree_manager.py"
    click pentestAgent "https://github.com/gh05tcrew/pentestagent/blob/main/core/pentest_agent.py"
    click pttReasoning "https://github.com/gh05tcrew/pentestagent/blob/main/core/ptt_reasoning.py"
    click modelManager "https://github.com/gh05tcrew/pentestagent/blob/main/core/model_manager.py"
    click knowledgeBase "https://github.com/gh05tcrew/pentestagent/blob/main/rag/knowledge_base.py"
    click embedding "https://github.com/gh05tcrew/pentestagent/blob/main/rag/embedding.py"
    click configureMCP "https://github.com/gh05tcrew/pentestagent/blob/main/tools/configure_mcp.py"
    click mcpManager "https://github.com/gh05tcrew/pentestagent/blob/main/tools/mcp_manager.py"
    click workflowDefs "https://github.com/gh05tcrew/pentestagent/blob/main/workflows/workflow_definitions.py"
    click workflowEngine "https://github.com/gh05tcrew/pentestagent/blob/main/workflows/workflow_engine.py"
    click reportGen "https://github.com/gh05tcrew/pentestagent/blob/main/reporting/generators.py"
    click main "https://github.com/gh05tcrew/pentestagent/blob/main/main.py"

    %% Styles
    classDef ui fill:#AED6F1,stroke:#1F618D,stroke-width:1px
    classDef config fill:#F9E79F,stroke:#B7950B,stroke-width:1px
    classDef configFile fill:#FDEBD0,stroke:#BA4A00,stroke-width:1px,stroke-dasharray: 5 5
    classDef core fill:#ABEBC6,stroke:#196F3D,stroke-width:1px
    classDef rag fill:#D2B4DE,stroke:#6C3483,stroke-width:1px
    classDef storage fill:#F5CBA7,stroke:#A04000,stroke-width:1px,shape:cylinder
    classDef tools fill:#F8C471,stroke:#B9770E,stroke-width:1px
    classDef workflow fill:#AED6F1,stroke:#1F618D,stroke-width:1px
    classDef report fill:#F5B7B1,stroke:#C0392B,stroke-width:1px
    classDef external fill:#D5D8DC,stroke:#566573,stroke-width:1px,shape:cloud
    classDef entry fill:#D6EAF8,stroke:#1A5276,stroke-width:1px,stroke-dasharray: 5 5
```
