# y-scope/log-surgeon architecture

> Architecture diagram of the GitHub repository y-scope/log-surgeon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/y-scope/log-surgeon
- Repository: https://github.com/y-scope/log-surgeon

## Overview

##### Project Type and Purpose
- Type: High-performance C++ library for parsing unstructured text logs.
- Primary use case: Users supply a schema of variable regexes; the library reads raw log streams, tokenizes and parses them into structured events.

##### Main Components
A. I/O Layer (Input/Output)
- FileReader: reads from disk streams
- BufferParser: reads from in-memory buffers
- ReaderParser: façade that wraps FileReader/BufferParser and exposes parse_next_event()

B. Schema Handling
- SchemaParser: parses a schema file (examples/schema.txt) into a Schema object
- Schema: holds variable names, regex ASTs, compiled automata

C. Lexical Layer
- Lexer: applies per-variable regex rules (LexicalRule) to produce tokens
- Token: stores a matched substring and metadata

D. Automata Engine
- RegexAST & ParserAst: represent regex grammar
- Nfa/NfaState/NfaTransition: constructs NFA from AST
- Dfa/DfaState/DfaTransition: converts NFA to DFA, optimizes variable matching
- PrefixTree, UnicodeIntervalTree: auxiliary structures for character classes

E. Parsing Layer
- Parser/Lalr1Parser: generic template-based parsers for schema grammar (LALR1)
- ParserInputBuffer/ParserOutputBuffer: manage input streams and parse results
- LogParser: orchestrates reading, lexing, automata matching, and event assembly

F. Data Model
- LogEvent & LogEventView: aggregate tokens into a structured event object
- UniqueIdGenerator, Constants: utilities for IDs and shared constants

G. Build & QA
- CMake (CMakeLists.txt + cmake/*.cmake)
- Tests (Catch2) under tests/
- Examples under examples/
- Linting tools & workflows (clang-tidy, Taskfiles)

##### Interactions & Data Flow
1. User code calls ReaderParser::reset_and_set_reader()
2. ReaderParser invokes FileReader or BufferParser to fetch raw bytes → feeds ParserInputBuffer
3. LogParser (via parse_next_event) triggers Lexer to scan input per-variable DFA tables
4. Lexer emits Token instances
5. Tokens are aggregated into a LogEvent (via LogParserOutputBuffer)
6. Client retrieves LogEventView for structured access

##### Architectural Patterns & Principles
- Layered modular design: clear separation of I/O, schema, lexing, parsing, data model
- Compiler-inspired pipeline: grammar → AST → NFA → DFA
- Template-based generic parsers (LALR1) for schema language
- Extension points: custom LexicalRule, plugging in new Automata handlers

c. In the center, Lexical & Automata: show RegexAST → NFA → DFA, all feeding into Lexer

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% I/O Layer
    subgraph "I/O Layer"
        direction TB
        FileReader["FileReader"]:::io
        BufferParser["BufferParser"]:::io
        ReaderParser["ReaderParser"]:::io
    end

    %% Schema Layer
    subgraph "Schema Layer"
        direction TB
        SchemaParser["SchemaParser"]:::schema
        Schema["Schema"]:::schema
        ExampleSchema["schema.txt"]:::schema
    end

    %% Lexical & Automata Layer
    subgraph "Lexical & Automata"
        direction TB
        RegexAST["RegexAST"]:::lex
        NFA["Nfa"]:::lex
        DFA["Dfa"]:::lex
        Lexer["Lexer (extensible)"]:::lex
    end

    %% Parsing Layer
    subgraph "Parsing Layer"
        direction TB
        ParserInputBuffer["ParserInputBuffer"]:::parse
        ParserOutputBuffer["LogParserOutputBuffer"]:::parse
        LogParser["LogParser"]:::parse
    end

    %% Data Model
    subgraph "Data Model"
        direction TB
        Token["Token"]:::data
        LogEvent["LogEvent"]:::data
        LogEventView["LogEventView"]:::data
    end

    %% Utilities
    subgraph "Utilities"
        direction TB
        UniqueIdGenerator["UniqueIdGenerator"]:::util
        Constants["Constants"]:::util
    end

    %% Flow
    FileReader -->|"raw bytes"| ParserInputBuffer
    BufferParser -->|"raw bytes"| ParserInputBuffer
    SchemaParser -->|"compile schema"| Schema
    Schema -->|"DFA tables"| DFA
    RegexAST -->|"to NFA"| NFA
    NFA -->|"to DFA"| DFA
    DFA -->|"match regex"| Lexer
    Lexer -->|"tokens"| ParserOutputBuffer
    ParserInputBuffer -->|input| LogParser
    ParserOutputBuffer -->|assemble events| LogParser
    LogParser -->|"LogEvent"| LogEvent
    LogEvent -->|view| LogEventView

    %% Click Events
    click FileReader "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/FileReader.hpp"
    click BufferParser "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/BufferParser.hpp"
    click ReaderParser "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/ReaderParser.hpp"
    click SchemaParser "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/SchemaParser.hpp"
    click Schema "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/Schema.hpp"
    click ExampleSchema "https://github.com/y-scope/log-surgeon/blob/main/examples/schema.txt"
    click Lexer "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/Lexer.hpp"
    click Token "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/Token.hpp"
    click RegexAST "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/finite_automata/RegexAST.hpp"
    click NFA "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/finite_automata/Nfa.hpp"
    click DFA "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/finite_automata/Dfa.hpp"
    click ParserInputBuffer "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/ParserInputBuffer.hpp"
    click ParserOutputBuffer "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/LogParserOutputBuffer.hpp"
    click LogParser "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/LogParser.hpp"
    click LogEvent "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/LogEvent.hpp"
    click UniqueIdGenerator "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/UniqueIdGenerator.hpp"
    click Constants "https://github.com/y-scope/log-surgeon/blob/main/src/log_surgeon/Constants.hpp"

    %% Styles
    classDef io fill:#cce5ff,stroke:#004085
    classDef schema fill:#d1ecf1,stroke:#0c5460
    classDef lex fill:#fff3cd,stroke:#856404
    classDef parse fill:#d4edda,stroke:#155724
    classDef data fill:#e2e3f3,stroke:#383d41
    classDef util fill:#f8f9fa,stroke:#6c757d
```
