How bsfishy/zig-lexer works
Main Components
• Lexer Core – Represented by files such as src/main.zig and the overall lexer module. This is the entry point that ties all components together.
• Regex/Pattern Parser – Shown in src/regex.zig (and possibly parts of the lexer module) that handles converting token pattern strings into a tree-like structure.
• Jump Table Generator – Located in src/jump_table.zig and related files (like src/array_list.zig, src/map.zig). It builds the jump table data structure from the parsed regex-tree. Emphasize that this component uses Zig’s compile-time (comptime) features.
• Token Processing – Where tokens are identified and built into an exhaustive enum, as well as handling skipping of tokens based on flags.
• Visualization API – The lexer also supports exporting its jump tables as Graphviz graphs (as demonstrated in the README). Although not a core processing component, it’s important for debugging and architectural understanding.
• Example/Test Integration – The file example.lang serves as an example input for the lexer to demonstrate its functionality.
Relationships and Data Flows
• From the parser, the hierarchical representation of patterns is passed into the Jump Table Generator.
• The Jump Table Generator compiles these into static, efficient tables via Zig’s comptime capabilities.
• The Lexer Core then uses these jump tables to iterate over the input text and produce tokens.
• Optionally, include a branch showing how the visualization API takes the in-memory jump tables and produces a Graphviz output.
Architectural Patterns and Design Principles
• Compile-Time Optimization – Emphasize the use of Zig’s comptime to generate efficient data structures (jump tables).
• Separation of Concerns – Different modules/files are used to parse patterns, build the jump table, and perform lexing at runtime.
• Data-Driven Architecture – The lexer is driven by a table (pattern/token mapping) that is generated dynamically during compilation.
• Extendability – The design allows for easily modifying or extending token patterns without needing to change the lexing engine.
Relevant Technologies and Technologies/Frameworks
• Zig Language – The entire project is implemented in Zig.
• Comptime Features – Used for generating jump tables and ensuring performance.
• Regular Expression-inspired Pattern Matching – Custom lightweight regex language tailored to the project’s needs.
• Graphviz – As a visualization aid for generated jump tables (integration shown in the README).