How itsamziii/tagparse works
Project type and purpose
• This is an open-source, Node.js library (“tagparse”) for high-performance, stream-based lexing and parsing of custom tag-based syntax.
• Its main goal is to let real-time and high-concurrency apps feed in large text streams (or strings), and extract/transform tokens (literals, variables, functions) into an AST or value stream with minimal memory overhead.
File structure
• src/index.ts
• Library entry-point that exports Lexer, Parser, types, etc.
• src/lib
• Errors.ts – custom error definitions
• Lexer.ts – implements the streaming lexer
• Stream.ts – a wrapper/adapter over Node.js Readable streams
• Parser/Parser.ts – orchestrates token consumption into AST nodes
• Parser/TagParsers – contains pluggable tag handlers:
– VariableTagParser.ts
– FunctionTagParser.ts
– TagParser.ts (base interface)
• src/types.ts – shared enums & interfaces (e.g., NodeType)
• src/utils.ts – helper functions
• tsup.config.ts – build & bundling config
• test/ – unit tests for lexer, strict mode, tag parsing
• .github/ – CI/Publish workflows (ci.yml, publish.yml)
README for additional insights
• Core runtime flow:
Input → Lexer (stream or string) → token stream → Parser → AST nodes → consumer logic (variable/function resolution)
• Configurable options: tagStart, tagEnd, parseTags, strict, plus custom functionParser & variableParser.
• Use cases: templating engines, chat systems, dashboards, real-time data processing.
a. Main components:
1. Consumer/Application – the calling code (e.g., a templating engine or real-time processor)
2. Input Source – can be a raw string or a Node.js Readable stream
3. Stream Adapter (Stream.ts) – normalizes input into an async iterable of chunks (optional step)
4. Lexer (Lexer.ts) – reads chunks, identifies tag boundaries & literal segments, emits tokens
5. Token Stream – asynchronous stream of tokens with types (literal, tagStart, tagContent, tagEnd)
6. Parser (Parser.ts) – consumes tokens, applies TagParsers to build AST nodes (Variable, Function, Literal)
7. TagParsers – pluggable handlers for each tag type (FunctionTagParser, VariableTagParser)
8. AST/Node List – output array of nodes, each carrying raw, type, children, etc.
9. Consumer Logic – application-specific code that takes the AST & resolves values (e.g., lookup variables, execute functions)
10. Error Handler – central Errors module to throw or recover from syntax issues (strict vs. non-strict)
b. Relationships & interactions:
• The Consumer injects input into the Input Source → Stream Adapter →
• Lexer pulls from Stream Adapter → Token Stream →
• Parser reads from Token Stream → calls appropriate TagParser →
• TagParser returns node fragments back to Parser → AST nodes → back to Consumer → final output.
• At each stage, Errors may be raised (strict mode).
c. Architectural patterns & design principles:
• Streaming Pipeline – non-blocking, back-pressure capable.
• Plugin Pattern – TagParsers implement a common interface for extensibility.
• Separation of Concerns – Lexer ≠ Parser ≠ Consumer logic.
• Async/Await & Iterators – flow control via async functions and iterators.
d. Key technologies and libraries:
• Native Node.js Streams (Readable)
• TypeScript for types and interfaces
• tsup for bundling
• GitHub Actions for CI and publishing
• Jest/Mocha (implied by .test.ts) for unit testing