# bsfishy/zig-lexer architecture

> Architecture diagram of the GitHub repository bsfishy/zig-lexer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bsfishy/zig-lexer
- Repository: https://github.com/bsfishy/zig-lexer

## Overview

##### Main Components
- Lexer Core – Represented by files such as src/main.zig and the overall lexer module. This is the entry point that ties all components together.
- Regex/Pattern Parser – Shown in src/regex.zig (and possibly parts of the lexer module) that handles converting token pattern strings into a tree-like structure.
- Jump Table Generator – Located in src/jump_table.zig and related files (like src/array_list.zig, src/map.zig). It builds the jump table data structure from the parsed regex-tree. Emphasize that this component uses Zig’s compile-time (comptime) features.
- Token Processing – Where tokens are identified and built into an exhaustive enum, as well as handling skipping of tokens based on flags.
- Visualization API – The lexer also supports exporting its jump tables as Graphviz graphs (as demonstrated in the README). Although not a core processing component, it’s important for debugging and architectural understanding.
- Example/Test Integration – The file example.lang serves as an example input for the lexer to demonstrate its functionality.

##### Relationships and Data Flows
- From the parser, the hierarchical representation of patterns is passed into the Jump Table Generator.
- The Jump Table Generator compiles these into static, efficient tables via Zig’s comptime capabilities.
- The Lexer Core then uses these jump tables to iterate over the input text and produce tokens.
- Optionally, include a branch showing how the visualization API takes the in-memory jump tables and produces a Graphviz output.

##### Architectural Patterns and Design Principles
- Compile-Time Optimization – Emphasize the use of Zig’s comptime to generate efficient data structures (jump tables).
- Separation of Concerns – Different modules/files are used to parse patterns, build the jump table, and perform lexing at runtime.
- Data-Driven Architecture – The lexer is driven by a table (pattern/token mapping) that is generated dynamically during compilation.
- Extendability – The design allows for easily modifying or extending token patterns without needing to change the lexing engine.

##### Relevant Technologies and Technologies/Frameworks
- Zig Language – The entire project is implemented in Zig.
- Comptime Features – Used for generating jump tables and ensuring performance.
- Regular Expression-inspired Pattern Matching – Custom lightweight regex language tailored to the project’s needs.
- Graphviz – As a visualization aid for generated jump tables (integration shown in the README).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Component
    E["Example/Test Integration (example.lang)"]:::input

    %% Runtime Components
    subgraph "Runtime Components"
        L["Lexer Core (src/main.zig & src/lexer.zig)"]:::runtime
        T["Token Processing (inside Lexer Core)"]:::runtime
        O["Tokens Output"]:::runtime
    end

    %% Compile-Time Components
    subgraph "Compile-Time Components"
        R["Regex/Pattern Parser (src/regex.zig)"]:::compileTime
        J["Jump Table Generator (src/jump_table.zig)"]:::compileTime
        A["Array List Module (src/array_list.zig)"]:::compileTime
        M["Map Module (src/map.zig)"]:::compileTime
    end

    %% Auxiliary Component
    V["Visualization API (README.md - Graphviz)"]:::aux

    %% Data Flow Connections
    E -->|"provides input"| L
    L -->|"parses patterns"| R
    R -->|"builds regex-tree"| J
    J -->|"includes"| A
    J -->|"includes"| M
    J -->|"generates jump tables (comptime)"| T
    T -->|"produces"| O
    J -->|"exports data for debugging"| V

    %% Click Events for Component Mapping
    click L "https://github.com/bsfishy/zig-lexer/blob/main/src/main.zig"
    click L "https://github.com/bsfishy/zig-lexer/blob/main/src/lexer.zig"
    click R "https://github.com/bsfishy/zig-lexer/blob/main/src/regex.zig"
    click J "https://github.com/bsfishy/zig-lexer/blob/main/src/jump_table.zig"
    click A "https://github.com/bsfishy/zig-lexer/blob/main/src/array_list.zig"
    click M "https://github.com/bsfishy/zig-lexer/blob/main/src/map.zig"
    click V "https://github.com/bsfishy/zig-lexer/blob/main/README.md"
    click E "https://github.com/bsfishy/zig-lexer/blob/main/example.lang"

    %% Styles
    classDef runtime fill:#ccffcc,stroke:#009933,stroke-width:2px;
    classDef compileTime fill:#ffe6b3,stroke:#ff9900,stroke-width:2px;
    classDef aux fill:#cce5ff,stroke:#0066cc,stroke-width:2px;
    classDef input fill:#f2f2f2,stroke:#666666,stroke-width:2px;
```
