# pzaino/thecrowler architecture

> Architecture diagram of the GitHub repository pzaino/thecrowler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pzaino/thecrowler
- Repository: https://github.com/pzaino/thecrowler

## Overview

##### Main Components

a) Core Services (represented as separate containers):
- Search API Service (services/api)
- Events Service (services/events)
- Crawler Engine (main application)
- PostgreSQL Database

b) Supporting Components:
- Virtual Desktop Infrastructure (VDI) for external tools
- Agent System (agents directory)
- Plugin System (plugins directory)
- Rules Engine (rules directory)

##### Key Architectural Elements

a) Package Organization (pkg directory):
- Agent Management (pkg/agent)
- Configuration Management (pkg/config)
- Crawler Core (pkg/crawler)
- Database Operations (pkg/database)
- Detection System (pkg/detection)
- Expression Interpreter (pkg/exprterpreter)
- Fingerprinting System (pkg/fingerprints)
- HTTP Information Gathering (pkg/httpinfo)
- Network Information Gathering (pkg/netinfo)
- Plugin System (pkg/plugin)
- Ruleset Engine (pkg/ruleset)

##### Frontend/API Layer
- Search API Service
- Events Service
- External API Clients

##### Processing Layer
- Crawler Engine
- Rules Engine
- Plugin System
- Agent System
- VDI Container

##### Data Layer
- PostgreSQL Database
- File Storage (for configurations and rules)

##### Key Interactions to Highlight

- API service interactions with the crawler engine
- Event service communication patterns
- Database access patterns
- Plugin and agent integration points
- VDI container interaction with external services
- Rule engine workflow

##### Additional Elements

- Notes for important configurations
- Security boundaries
- Scalable components
- External integration points

##### Technical Details

- Network ports and protocols
- Key configuration points
- Database partitioning strategy
- Security measures (NET_ADMIN, NET_RAW capabilities)
- Service discovery mechanisms
- Health check endpoints

- Low-noise crawling architecture
- Customizable crawling patterns
- Scope variability implementation
- Detection capabilities
- Ruleset system integration
- Search engine functionality

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and classes
    classDef api fill:#f9f,stroke:#333,stroke-width:2px
    classDef service fill:#afd,stroke:#333,stroke-width:2px
    classDef database fill:#ddf,stroke:#333,stroke-width:2px
    classDef engine fill:#fda,stroke:#333,stroke-width:2px
    classDef system fill:#dfd,stroke:#333,stroke-width:2px
    classDef component fill:#ffd,stroke:#333,stroke-width:2px

    %% Frontend/API Layer
    subgraph Frontend
        API[Search API Service]:::api
        Events[Events Service]:::api
        ExtAPI[External API Clients]:::api
    end

    %% Processing Layer
    subgraph Processing
        CrawlerEngine[Crawler Engine]:::engine
        
        subgraph CoreSystems
            AgentSys[Agent System]:::system
            PluginSys[Plugin System]:::system
            RulesEngine[Rules Engine]:::system
        end
        
        subgraph Components
            Config[Configuration Management]:::component
            Detection[Detection System]:::component
            ExprInt[Expression Interpreter]:::component
            Fingerprint[Fingerprinting System]:::component
            HTTPInfo[HTTP Info Gathering]:::component
            NetInfo[Network Info Gathering]:::component
        end
        
        VDI[Virtual Desktop Infrastructure]:::system
    end

    %% Data Layer
    subgraph Data
        DB[(PostgreSQL Database)]:::database
        FileStore[(File Storage)]:::database
    end

    %% Relationships
    API --> CrawlerEngine
    Events --> CrawlerEngine
    ExtAPI --> API
    
    CrawlerEngine --> AgentSys
    CrawlerEngine --> PluginSys
    CrawlerEngine --> RulesEngine
    
    AgentSys --> VDI
    PluginSys --> VDI
    
    CrawlerEngine --> Components
    Components --> DB
    RulesEngine --> FileStore
    
    %% Click events for component mapping
    click API "https://github.com/pzaino/thecrowler/tree/main/services/api/"
    click Events "https://github.com/pzaino/thecrowler/tree/main/services/events/"
    click CrawlerEngine "https://github.com/pzaino/thecrowler/tree/main/pkg/crawler/"
    click DB "https://github.com/pzaino/thecrowler/blob/main/pkg/database/postgresql-setup-v1.5.pgsql"
    click AgentSys "https://github.com/pzaino/thecrowler/tree/main/pkg/agent/"
    click PluginSys "https://github.com/pzaino/thecrowler/tree/main/pkg/plugin/"
    click RulesEngine "https://github.com/pzaino/thecrowler/tree/main/pkg/ruleset/"
    click Config "https://github.com/pzaino/thecrowler/tree/main/pkg/config/"
    click Detection "https://github.com/pzaino/thecrowler/tree/main/pkg/detection/"
    click ExprInt "https://github.com/pzaino/thecrowler/tree/main/pkg/exprterpreter/"
    click Fingerprint "https://github.com/pzaino/thecrowler/tree/main/pkg/fingerprints/"
    click HTTPInfo "https://github.com/pzaino/thecrowler/tree/main/pkg/httpinfo/"
    click NetInfo "https://github.com/pzaino/thecrowler/tree/main/pkg/netinfo/"
    click VDI "https://github.com/pzaino/thecrowler/tree/main/selenium-patches/"
```
