How projectdiscovery/httpx works
Main Components Identification
The system should be divided into these primary components:
a. Command Line Interface (CLI)
• Located in cmd/httpx
• Handles user input and flag parsing
• Manages configuration and output formatting
b. Core Engine (Runner)
• Located in runner/
• Orchestrates the HTTP probing operations
• Manages concurrent execution and rate limiting
c. HTTP Processing Module
• Located in common/httpx/
• Handles HTTP/HTTPS requests
• Manages protocol detection and fallback logic
d. Utility Modules
• Located in common/
• Custom extractors (customextract/)
• Header processing (customheader/)
• File operations (fileutil/)
• Hash calculations (hashes/)
Component Relationships
[CLI Layer] → [Runner Layer] → [HTTP Processing Layer]
• ↓
• [Utility Modules]
Specific Elements
a. Input Processing
• Input sources (URLs, hosts, CIDR)
• File input handling
• Raw request processing
b. Core Processing Pipeline
• Request generation
• Concurrent execution
• Rate limiting
• Retry mechanism
c. Feature Modules
• TLS/SSL handling
• HTTP2 support
• Pipeline processing
• Virtual host detection
• Screenshot capability (headless browser)
d. Output Handling
• Multiple format support (JSON, CSV)
• Response storage
• Custom output formatting
Important Architectural Patterns to Highlight
a. Pipeline Pattern
• Flow from input → processing → output
• Concurrent processing capability
b. Plugin Architecture
• How different probes can be enabled/disabled
• Extensibility of the system
c. Configuration Management
• How different options affect the processing pipeline
• Where configuration is applied
Technical Details
• Rate limiting mechanisms
• Concurrent processing with goroutines
• Protocol handling (HTTP/HTTPS/HTTP2)
• Security features (TLS, custom headers)
• Integration points with external services
┌─ Input Section ─┐ ┌─ Processing Section ─┐ ┌─ Output Section ─┐
• File Input │ → │ - HTTP Processing │ → │ - JSON Output │
• URL Input │ │ - TLS Handling │ │ - CSV Output │
• CIDR Input │ │ - Probe Execution │ │ - File Storage │
• ┘ └───────────────────────┘ └────────────────┘