How pzaino/thecrowler works
Main Components
a) Core Services (represented as separate containers):
• Search API Service (services/api)
• Events Service (services/events)
• Crawler Engine (main application)
• PostgreSQL Database
b) Supporting Components:
• Virtual Desktop Infrastructure (VDI) for external tools
• Agent System (agents directory)
• Plugin System (plugins directory)
• Rules Engine (rules directory)
Key Architectural Elements
a) Package Organization (pkg directory):
• Agent Management (pkg/agent)
• Configuration Management (pkg/config)
• Crawler Core (pkg/crawler)
• Database Operations (pkg/database)
• Detection System (pkg/detection)
• Expression Interpreter (pkg/exprterpreter)
• Fingerprinting System (pkg/fingerprints)
• HTTP Information Gathering (pkg/httpinfo)
• Network Information Gathering (pkg/netinfo)
• Plugin System (pkg/plugin)
• Ruleset Engine (pkg/ruleset)
Frontend/API Layer
• Search API Service
• Events Service
• External API Clients
Processing Layer
• Crawler Engine
• Rules Engine
• Plugin System
• Agent System
• VDI Container
Data Layer
• PostgreSQL Database
• File Storage (for configurations and rules)
Key Interactions to Highlight
• API service interactions with the crawler engine
• Event service communication patterns
• Database access patterns
• Plugin and agent integration points
• VDI container interaction with external services
• Rule engine workflow
Additional Elements
• Notes for important configurations
• Security boundaries
• Scalable components
• External integration points
Technical Details
• Network ports and protocols
• Key configuration points
• Database partitioning strategy
• Security measures (NET_ADMIN, NET_RAW capabilities)
• Service discovery mechanisms
• Health check endpoints
• Low-noise crawling architecture
• Customizable crawling patterns
• Scope variability implementation
• Detection capabilities
• Ruleset system integration
• Search engine functionality