How tree-sitter/py-tree-sitter works
Project Type and Purpose
• This is a Python library providing bindings to the Tree-sitter parsing engine (a C library).
• It is not a full-stack web app—it’s an extensible parser library that lets Python code invoke a high-performance C parser.
Main Components
a. Python API Layer
– Files under tree_sitter/ (__init__.py, .pyi) define classes: Language, Parser, Node, Tree, TreeCursor, Query, etc.
– These are what users import and call in their Python scripts or applications.
b. C Extension / Binding Layer
– Under tree_sitter/binding/*.c and types.h sits the CPython extension code.
– Exposes the Tree-sitter C API to Python by wrapping functions in Python-callable methods.
c. Tree-sitter Core
– In tree_sitter/core/ (and linked from submodules) lives the actual parser implementation and runtime.
– Compiled as a static or shared library that the binding layer links against.
d. Language Plug-ins
– External language libraries (e.g. tree-sitter-python wheel) provide language() functions.
– Loaded at runtime into the Language class.
e. Support Infrastructure
– Examples (examples/usage.py, examples/walk_tree.py) show how to use the API.
– Tests (tests/) validate functionality of each class.
– Docs (docs/) generated by Sphinx, illustrating API reference and guides.
– Packaging/config (pyproject.toml, setup.py, GitHub workflows) for CI, publishing, linting.
Relationships and Data Flow
• User Code → calls Python API Layer methods (e.g. Parser.parse()).
• Python API Layer → calls into C Extension Layer (via CPython C-API functions).
• C Extension Layer → invokes Tree-sitter Core functions to parse source text.
• Tree-sitter Core → returns opaque parse trees to the C Extension.
• C Extension Layer → wraps core structures into Python objects (Tree, Node, etc.) and returns them.
• Python Code then inspects or mutates these objects, or applies Query patterns.
Architectural Patterns and Principles
– Layered Architecture: clear separation between user-facing Python classes and low-level C bindings.
– Plugin/Extension Model: languages are separate wheels and loaded dynamically.
– Test-Driven: each core component has corresponding unit tests.
– Documentation-First: Sphinx docs mirror the package structure for discoverability.
Essential Elements
– Component labels: e.g., “TreeCursor (Python)”, “lookahead_iterator.c”