How phpstan/phpdoc-parser works
This project is a PHP library whose main purpose is to parse PHPDoc blocks into an Abstract Syntax Tree (AST) and enable modifications (including format-preserving printing) of PHPDoc comments. It serves as a core component used primarily by PHPStan for advanced PHPDoc handling, though it also supports Doctrine Annotations.
Main Components
• Lexer Component: Located in src/Lexer, this component tokenizes PHPDoc comments into tokens.
• Parser Components:
– PhpDocParser (in src/Parser) that orchestrates parsing.
– ConstExprParser and TypeParser, which handle constant expressions and types respectively.
– TokenIterator that processes the tokens produced by the Lexer.
• AST Representation:
– A robust hierarchy of AST nodes in the src/Ast directory (including specialized nodes for PHPDoc handling, types like IdentifierTypeNode, union/intersection types, annotations, and PHPDoc tags).
• Node Traversal/Visitors:
– The NodeTraverser and NodeVisitor (e.g., CloningVisitor within src/Ast/NodeVisitor) enable manipulations over the AST.
• Printer Component:
– Found in src/Printer, responsible for printing the AST back to a PHPDoc string while preserving the original formatting when possible.
• Configuration:
– ParserConfig (in src/ParserConfig.php) which is used to enable specific lexer attributes (such as lines, indexes, comments).
• Auxiliary/Support Components:
– The “apigen” directory for generating API documentation/themes.
– Documentation files (in the doc directory) describing the grammar rules and supported constructs.
• Testing Framework:
– A comprehensive test suite under tests which validates the parsing, AST generation, traversal, and printer functionalities.
Relationships and Data Flow
• Input Flow: A PHPDoc string is first passed to the Lexer, which tokenizes the input.
• Parsing Flow: Tokens are fed into the TokenIterator. The Parser (PhpDocParser) works with the TypeParser and ConstExprParser to generate an AST representing the PHPDoc structure.
• Transformation Flow: The AST can be traversed and manipulated using NodeTraverser with various NodeVisitor implementations—for example, cloning the AST before modifications.
• Output Flow: The Printer component uses the modified AST along with the original tokens (for format preservation) to generate an updated PHPDoc comment.
• Configuration Flow: The ParserConfig influences the behavior of both the Lexer and the Parser, determining which attributes to use during tokenization and parsing.
Architectural Patterns and Principles
• Layered Architecture:
– Tokenization, parsing, AST representation, AST manipulation, and printing are structured in separate but interacting layers.
• Separation of Concerns:
– Each module is dedicated to a specific task (e.g., lexing, parsing, printing), making the system modular and easier to test.
• Extensibility:
– The design allows for easy extension by adding new AST node types or parser constructs (for instance, to support new PHPDoc features or additional annotation styles).
• Similarities to Compiler Architecture:
– Input (document) → Lexical Analysis → Parsing → AST → Transformations → Code Generation (or, in this case, re-printing).