How josh1643/totalrecon works
Project type and purpose
• This is an open-source Python library with an optional CLI.
• Purpose: passive reconnaissance—scanning text or PDF inputs for subdomains, emails, S3 buckets, then using a locally hosted AI model to summarize findings.
File structure
• Top level
• cli.py – CLI entrypoint, parses arguments, invokes library APIs, prints results
• setup.py / pyproject.toml – packaging and dependency declarations
• totalrecon/ package
• extract.py – core extraction logic (regex‐based domain, email, bucket discovery)
• cert.py – specialized certificate parsing or validation functions
• ai.py – wraps the Hugging Face “flan-t5-small” summarization pipeline
• No separate “frontend/backend” folders—monolithic library exposing both CLI and API.
README for additional insights
• Describes two primary workflows: extraction and AI summarization
• External dependency: Hugging Face Hub for model download (cached locally on first use)
• Intended users: red teamers, bug bounty hunters, CTF players—focus on offline, lightweight operation
Components
a. User Interface Layer
• CLI (cli.py) and Python library import interface
b. Extraction Engine
• extract.py (text/PDF ingestion module + regex scanners)
• cert.py (certificate‐specific parsing)
c. AI Summarization Engine
• ai.py (initializes and invokes the Transformers pipeline)
• Local Model Cache (on-disk storage managed by Hugging Face SDK)
• Hugging Face Hub (remote model repository)
d. Data Flow and Artifacts
• Input: plaintext files, PDF files, raw strings
• Output: JSON structure with “domains”, “emails”, “s3_buckets”, “recon_summaries”
Relationships and data flow
• User invokes CLI or library call →
Extraction Engine parses input →
Returns raw findings →
AI Engine loads (or reuses cached) model →
Generates human‐readable summary →
CLI/API surfaces combined JSON to user
Architectural patterns and design principles
• Modular monolith: clear separation between I/O (CLI), domain logic (extract/cert), and AI services
• Lazy loading / caching: model is downloaded once, reused offline thereafter
• Single responsibility: each module focuses on one aspect of the workflow
Detail
• Important functions (e.g., “extract_from_text” in extract.py, “summarize” in ai.py)
• Config files (pyproject.toml) as part of deployment/package management
• Error-handling or validation flow if desired (e.g., PDF parse failures)