How y-scope/log-surgeon works
Project Type and Purpose
• Type: High-performance C++ library for parsing unstructured text logs.
• Primary use case: Users supply a schema of variable regexes; the library reads raw log streams, tokenizes and parses them into structured events.
Main Components
A. I/O Layer (Input/Output)
• FileReader: reads from disk streams
• BufferParser: reads from in-memory buffers
• ReaderParser: façade that wraps FileReader/BufferParser and exposes parse_next_event()
B. Schema Handling
• SchemaParser: parses a schema file (examples/schema.txt) into a Schema object
• Schema: holds variable names, regex ASTs, compiled automata
C. Lexical Layer
• Lexer: applies per-variable regex rules (LexicalRule) to produce tokens
• Token: stores a matched substring and metadata
D. Automata Engine
• RegexAST & ParserAst: represent regex grammar
• Nfa/NfaState/NfaTransition: constructs NFA from AST
• Dfa/DfaState/DfaTransition: converts NFA to DFA, optimizes variable matching
• PrefixTree, UnicodeIntervalTree: auxiliary structures for character classes
E. Parsing Layer
• Parser/Lalr1Parser: generic template-based parsers for schema grammar (LALR1)
• ParserInputBuffer/ParserOutputBuffer: manage input streams and parse results
• LogParser: orchestrates reading, lexing, automata matching, and event assembly
F. Data Model
• LogEvent & LogEventView: aggregate tokens into a structured event object
• UniqueIdGenerator, Constants: utilities for IDs and shared constants
G. Build & QA
• CMake (CMakeLists.txt + cmake/*.cmake)
• Tests (Catch2) under tests/
• Examples under examples/
• Linting tools & workflows (clang-tidy, Taskfiles)
Interactions & Data Flow
1. User code calls ReaderParser::reset_and_set_reader()
2. ReaderParser invokes FileReader or BufferParser to fetch raw bytes → feeds ParserInputBuffer
3. LogParser (via parse_next_event) triggers Lexer to scan input per-variable DFA tables
4. Lexer emits Token instances
5. Tokens are aggregated into a LogEvent (via LogParserOutputBuffer)
6. Client retrieves LogEventView for structured access
Architectural Patterns & Principles
• Layered modular design: clear separation of I/O, schema, lexing, parsing, data model
• Compiler-inspired pipeline: grammar → AST → NFA → DFA
• Template-based generic parsers (LALR1) for schema language
• Extension points: custom LexicalRule, plugging in new Automata handlers
c. In the center, Lexical & Automata: show RegexAST → NFA → DFA, all feeding into Lexer