How gh05tcrew/pentestagent works
Project Type and Purpose
• GHOSTCREW is a command-line AI “red team” assistant (open-source tool) built in Python.
• It uses Large Language Models (via OpenAI API), a Retrieval-Augmented Generation (RAG) knowledge base, and MCP protocol to call external pentesting tools.
• Its main functions are natural-language driven pen-testing workflows, autonomous agent mode, interactive tool configuration, and markdown report generation.
File Structure
Top-level directories and roles:
• config/: application and constant definitions (app_config.py, constants.py).
• core/: the heart of the assistant:
• agent_mode_controller.py, agent_runner.py – orchestrate autonomous agents with task trees
• model_manager.py – wraps OpenAI API calls, streaming, model parameters
• pentest_agent.py, ptt_reasoning.py – decision-making logic
• task_tree_manager.py – builds/manages Pentesting Task Trees (PTT)
• rag/: RAG subsystem
• knowledge_base.py – loads text files, vector store abstraction
• embedding.py – handles embedding calls to LLM API
• tools/: MCP tooling integration
• configure_mcp.py, mcp_manager.py – read/write mcp.json, launch/communicate with MCP servers
• workflows/: predefined, repeatable pentesting sequences
• workflow_definitions.py, workflow_engine.py
• reporting/: markdown report generation (generators.py)
• ui/: CLI interface
• menu_system.py – top-level menus (chat vs workflow vs agent)
• conversation_manager.py – handles user I/O, multi-turn contexts
• knowledge/: optional RAG content (wordlist.txt, plus user-supplied files)
• main.py: application entrypoint loading config, kicking off UI
Key README Insights
• Modes: Chat, Workflows, Agent
• MCP server integrations (Node.js or Python backends) configured via mcp.json
• Knowledge-base enhancement optional, triggered at startup
• Streaming responses, conversation history, multi-line input
1. CLI UI Layer
– menu_system.py
– conversation_manager.py
2. Configuration Layer
– config/app_config.py, constants.py
– mcp.json (persistence)
3. Core Logic Layer
– Agent Mode Controller & Runner
– Task Tree Manager
– Pentest Agent & Reasoning
– Model Manager (LLM interface)
4. RAG Subsystem (optional path)
– Knowledge Base Loader
– Embedding Service
– Local vector store (in-memory or disk)
5. Tools Integration Layer
– MCP Manager
– Configure MCP Tool script
– External MCP Servers (node- or python-based tool daemons)
6. Workflows Engine
– Workflow Definitions
– Workflow Engine
7. Reporting Layer
– Markdown Report Generator
8. External Services
– OpenAI API (LLM & Embeddings)
– MCP Tool Servers (nmap, metasploit, ffuf, etc.)
• UI → Core Logic (user commands mapped to controllers)
• Core Logic → Model Manager → OpenAI API (prompts, responses)
• Core Logic (when RAG enabled) → RAG Subsystem → Model Manager → OpenAI API
• Core Logic/Workflows → MCP Manager → External MCP Servers → Results → Core Logic
• Core Logic/Workflows → Reporting → Markdown File Output
• Persistent Config (mcp.json, config files) read at startup & updated by Tools layer
C. Architectural Patterns & Principles
– Layered architecture (UI / Application / Integration / External)
– Plugin/Adapter pattern for MCP tools (mcp_manager abstracts heterogeneous tool servers)
– Strategy pattern in workflows (each workflow definition encapsulates a sequence of tool calls)
– Pipeline for RAG (load corpus → embed → vector query → prompt enrichment)
– Agent-based control for autonomous pentesting (task trees + reasoning)
D. Key Technologies & Frameworks
– Python 3, standard libraries
– OpenAI API (chat completions + embeddings)
– Node.js or Python daemons for MCP tools (via npx or local servers)
– JSON config (mcp.json)
– CLI I/O handling (built-in input/print, optional streaming)
Detail, Not Overthinking
– Break the system into as many logical components as above.
– Capture conditional paths (RAG vs pure LLM chat, agent vs workflow vs chat).