# intel/hyperscan architecture

> Architecture diagram of the GitHub repository intel/hyperscan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/intel/hyperscan
- Repository: https://github.com/intel/hyperscan

## Overview

##### Main Components (Top Level)
a. API Layer (Top)
b. Compiler Layer (Middle)
c. Runtime Layer (Bottom)

##### Compiler Components (Middle Layer)
- Parser System (/src/parser/)
- Component handling
- Pattern parsing
- UTF-8 validation
- NFAGraph System (/src/nfagraph/)
- Pattern analysis
- Graph optimization
- Component transformation
- Rose Engine (/src/rose/)
- Pattern matching engine
- Build system
- Runtime matcher
- FDR Engine (/src/fdr/)
- Fast pattern matching
- Literal matching optimization

##### Runtime Components (Bottom Layer)
Essential runtime elements:
- Database Management (/src/database.*)
- Scratch Space Management (/src/scratch.*)
- Stream Handling (/src/rose/stream.*)
- SOM (Start of Match) Runtime (/src/som/)

##### Supporting Systems (Side Panels)
Create side containers for:
- Utility Functions (/src/util/)
- Platform-Specific Optimizations (/src/util/simd_*)
- Memory Management (/src/alloc.*)

- Pattern compilation flow: API → Parser → NFAGraph → Rose/FDR
- Runtime matching flow: API → Database → Runtime Engines
- Stream data flow through the runtime system

##### Integration Points
Interfaces for:
- PCRE compatibility layer
- Chimera subsystem integration
- External tool integration

##### Performance Features
Specialized components:
- Hardware acceleration support
- Pattern optimization paths
- Memory management optimization

Use these visual elements:
- Rectangles: Major components
- Rounded rectangles: APIs and interfaces
- Hexagons: Processing engines
- Diamonds: Decision points
- Labels: Clear component names and versions

##### Memory management strategy

Keep the abstraction level high enough to show the overall architecture while including sufficient detail to understand the pattern matching pipeline and runtime execution flow.

This design reflects Hyperscan's nature as a high-performance regex matching library with distinct compilation and runtime phases, optimized for processing large numbers of patterns simultaneously across data streams.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Style definitions
    classDef api fill:#e1f5fe,stroke:#01579b
    classDef compiler fill:#e8f5e9,stroke:#1b5e20
    classDef runtime fill:#fff3e0,stroke:#e65100
    classDef util fill:#f3e5f5,stroke:#4a148c
    classDef engine fill:#ffebee,stroke:#b71c1c

    %% API Layer
    subgraph API
        API[API Interface]:::api
    end

    %% Compiler Layer
    subgraph Compiler
        Parser["Parser System"]:::compiler
        NFAGraph["NFAGraph System"]:::compiler
        RoseEngine["Rose Engine"]:::engine
        FDREngine["FDR Engine"]:::engine

        %% Parser Components
        ParserComp["Pattern Parser"]:::compiler
        UTF8Valid["UTF-8 Validator"]:::compiler

        %% NFAGraph Components
        PatternAnalysis["Pattern Analysis"]:::compiler
        GraphOpt["Graph Optimization"]:::compiler

        %% Engine Components
        RoseBuild["Rose Build System"]:::engine
        RoseRuntime["Rose Runtime"]:::engine
        FDRMatcher["FDR Matcher"]:::engine
    end

    %% Runtime Layer
    subgraph Runtime
        DB["Database Management"]:::runtime
        Scratch["Scratch Space"]:::runtime
        Stream["Stream Handler"]:::runtime
        SOM["SOM Runtime"]:::runtime
        RuntimeCore["Runtime Core"]:::runtime
    end

    %% Supporting Systems
    subgraph Utils
        UtilFunc["Utility Functions"]:::util
        PlatformOpt["Platform Optimizations"]:::util
        MemoryMgmt["Memory Management"]:::util
    end

    %% Relationships
    API --> Parser
    Parser --> ParserComp
    Parser --> UTF8Valid
    ParserComp --> NFAGraph
    NFAGraph --> PatternAnalysis
    NFAGraph --> GraphOpt
    PatternAnalysis --> RoseEngine
    PatternAnalysis --> FDREngine
    RoseEngine --> RoseBuild
    RoseEngine --> RoseRuntime
    FDREngine --> FDRMatcher
    RoseRuntime --> DB
    FDRMatcher --> DB
    DB --> RuntimeCore
    RuntimeCore --> Stream
    RuntimeCore --> SOM
    RuntimeCore --> Scratch

    %% Utility Connections
    UtilFunc -.-> Parser
    UtilFunc -.-> NFAGraph
    PlatformOpt -.-> RoseEngine
    PlatformOpt -.-> FDREngine
    MemoryMgmt -.-> DB
    MemoryMgmt -.-> Scratch

    %% Click Events
    click Parser "https://github.com/intel/hyperscan/tree/master//src/parser/"
    click ParserComp "https://github.com/intel/hyperscan/blob/master//src/parser/Parser.h"
    click UTF8Valid "https://github.com/intel/hyperscan/blob/master//src/parser/utf8_validate.h"
    click NFAGraph "https://github.com/intel/hyperscan/tree/master//src/nfagraph/"
    click PatternAnalysis "https://github.com/intel/hyperscan/blob/master//src/nfagraph/ng.h"
    click GraphOpt "https://github.com/intel/hyperscan/blob/master//src/nfagraph/ng_misc_opt.h"
    click RoseEngine "https://github.com/intel/hyperscan/tree/master//src/rose/"
    click RoseBuild "https://github.com/intel/hyperscan/blob/master//src/rose/rose_build.h"
    click RoseRuntime "https://github.com/intel/hyperscan/blob/master//src/rose/runtime.h"
    click FDREngine "https://github.com/intel/hyperscan/tree/master//src/fdr/"
    click FDRMatcher "https://github.com/intel/hyperscan/blob/master//src/fdr/fdr.h"
    click DB "https://github.com/intel/hyperscan/blob/master//src/database.h"
    click Scratch "https://github.com/intel/hyperscan/blob/master//src/scratch.h"
    click Stream "https://github.com/intel/hyperscan/blob/master//src/rose/stream.c"
    click SOM "https://github.com/intel/hyperscan/tree/master//src/som/"
    click UtilFunc "https://github.com/intel/hyperscan/tree/master//src/util/"
    click PlatformOpt "https://github.com/intel/hyperscan/blob/master//src/util/simd_utils.h"
    click MemoryMgmt "https://github.com/intel/hyperscan/blob/master//src/util/alloc.h"
    click RuntimeCore "https://github.com/intel/hyperscan/blob/master//src/runtime.c"
```
