How elkarouh/hparsec works
What this repository is
Adascript is a source-to-source compiler for a Python-like language. The same .ady input is parsed once and then emitted into two target runtimes: Python 3 and Nim. The repo is organized around that split-brain backend design rather than around an application domain.
Top-level pipeline
The main data flow is: Adascript source → tokenizer → parser/grammar → AST-like node objects → backend-specific code generation. The two entrypoints, TO_PYTHON/py2py.py and TO_NIM/py2nim.py, drive that pipeline and differ only in the final emission target and some Nim-specific build orchestration.
Lexing and source normalization
hek_tokenize.py is the front door for source text. It does more than tokenization: it preprocesses Ada-style tick attributes like Type'Attr, preserves comment/newline structure with RichNL, and normalizes newline handling inside brackets. hek_helpers.py provides shared indentation and newline utilities used across the parser and emitters.
Parser combinator core
hek_parsec.py is the parser engine underneath the grammar. It provides combinator-style parsing primitives, packrat memoization, forward references, and parser-state management. This is the foundation that lets the language grammar be expressed declaratively instead of with hand-written recursive descent code.
Language grammar definition
ADASCRIPT_GRAMMAR/ contains the language-neutral grammar split by concern: expressions, simple statements, compound statements, and declarations. These modules define the syntax of Adascript itself, including Python compatibility plus additive features such as type declarations, ranges, case/when, shell statements, and shell variables.
AST-to-target transformation boundary
The grammar nodes are not the final output; each node type gets target-specific to_py() or to_nim() methods attached in the backend packages. This is the key architectural boundary: the same parsed structure is lowered differently depending on whether the output is Python or Nim. The backend modules mirror the grammar split with expression, statement, parser, and declaration emitters.
Python backend
TO_PYTHON/ translates Adascript into valid Python 3 source. It maps Adascript types to Python typing constructs, expands enums/dataclasses/variant records into Python class forms, rewrites shell and bash-style features into Python stdlib calls, and preserves Python-native semantics where possible. The backend is designed to stay close to idiomatic Python output, not just syntactic equivalence.
Nim backend
TO_NIM/ performs the analogous lowering into Nim. It adds Nim-specific constructs such as enum-indexed arrays, objects vs. ref objects, Nim case/of, and Nim runtime/import mapping. py2nim.py also contains build/runtime orchestration beyond transpilation, including incremental rebuild checks, cache management, and optional compile-and-run behavior.
Runtime and interoperability support
The generated code relies on different runtime assumptions in each target. Python output leans on the standard library and typing/dataclasses; Nim output sometimes needs extra support from a local BENCHMARK/stdlib.nim shim for Python-like conveniences such as queues and helper abstractions. The repo also handles Python-to-Nim interop through nimpy when a Python library has no direct Nim equivalent.
Main entry points: hek_py3_parser.py Py parser · hek_nim_parser.py Nim parser · hek_parsec.py Parsec core · py2nim.py py2nim
Components 21
Frontend
Source handling and language grammar
- Tokenizer · Front door for .ady text; handles tokenization, tick attributes, rich newlines, and bracket-aware normalization.
hek_tokenize.py - Helpers · Indentation and newline helpers used across parsing and emission.
hek_helpers.py - Expr grammar · Expression syntax and precedence rules for Adascript.
ADASCRIPT_GRAMMAR/py3expr.py - Stmt grammar · Simple statement syntax for the shared language grammar.
ADASCRIPT_GRAMMAR/py3stmt.py - Compound grammar · Compound statements, control flow, and block forms.
ADASCRIPT_GRAMMAR/py3compound_stmt.py - Decl grammar · Declarations, type forms, and other top-level language constructs.
ADASCRIPT_GRAMMAR/py_declarations.py
Parse core
Shared parser machinery
- Parsec core · Combinator parser framework with memoization, forward refs, and parser state.
hek_parsec.py
Backends
Target-specific lowering and build drivers
- Py parser · Python-target parser glue that attaches emission methods to grammar nodes.
TO_PYTHON/hek_py3_parser.py - Py expr · Expression lowering for Python 3 output.
TO_PYTHON/hek_py3_expr.py - Py stmt · Statement lowering for Python 3 output.
TO_PYTHON/hek_py3_stmt.py - Py decl · Declaration and type lowering for Python output.
TO_PYTHON/hek_py_declarations.py - py2py · Command-line driver that runs the parse-to-Python pipeline.
TO_PYTHON/py2py.py - Nim parser · Nim-target parser glue that connects grammar nodes to Nim emission.
TO_NIM/hek_nim_parser.py - Nim expr · Expression lowering for Nim output.
TO_NIM/hek_nim_expr.py - Nim stmt · Statement lowering for Nim output.
TO_NIM/hek_nim_stmt.py - Nim decl · Declaration and type lowering for Nim output.
TO_NIM/hek_nim_declarations.py - py2nim · Command-line driver plus incremental build, cache, and optional compile-and-run orchestration.
TO_NIM/py2nim.py - Nim stdlib · Local Nim support library for Python-like conveniences and helper abstractions.
TO_NIM/stdlib.nim
Validation
Benchmarks and regression coverage
- Benchmarks · Adascript programs and reference outputs that exercise major language features and workloads.
BENCHMARK - Nim tests · Focused backend and import/type regression cases for Nim output.
TO_NIM/TEST - Py tests · Focused Python backend regression checks.
TO_PYTHON/test.py
Connections 29
- Tokenizer → Parsec core: tokens · Source text is normalized and tokenized before parsing.
- Helpers → Tokenizer: newline/indent · Shared formatting utilities support tokenization and parsing.
- Helpers → Py stmt: formatting · Python emission uses shared indentation and newline helpers.
- Helpers → Nim stmt: formatting · Nim emission uses shared indentation and newline helpers.
- Parsec core → Expr grammar: drives · Parser combinators are used to express expression grammar.
- Parsec core → Stmt grammar: drives · Parser combinators are used to express statement grammar.
- Parsec core → Compound grammar: drives · Parser combinators are used to express compound statement grammar.
- Parsec core → Decl grammar: drives · Parser combinators are used to express declaration grammar.
- Expr grammar → Py parser: AST nodes · Grammar nodes are extended with Python emission behavior.
- Stmt grammar → Py parser: AST nodes · Grammar nodes are extended with Python emission behavior.
- Compound grammar → Py parser: AST nodes · Grammar nodes are extended with Python emission behavior.
- Decl grammar → Py parser: AST nodes · Grammar nodes are extended with Python emission behavior.
- Expr grammar → Nim parser: AST nodes · Grammar nodes are extended with Nim emission behavior.
- Stmt grammar → Nim parser: AST nodes · Grammar nodes are extended with Nim emission behavior.
- Compound grammar → Nim parser: AST nodes · Grammar nodes are extended with Nim emission behavior.
- Decl grammar → Nim parser: AST nodes · Grammar nodes are extended with Nim emission behavior.
- Py parser → Py expr: emits · Python parser glue dispatches to expression lowering.
- Py parser → Py stmt: emits · Python parser glue dispatches to statement lowering.
- Py parser → Py decl: emits · Python parser glue dispatches to declaration lowering.
- Nim parser → Nim expr: emits · Nim parser glue dispatches to expression lowering.
- Nim parser → Nim stmt: emits · Nim parser glue dispatches to statement lowering.
- Nim parser → Nim decl: emits · Nim parser glue dispatches to declaration lowering.
- py2py → Py parser: pipeline · Python driver runs the shared parse-to-Python flow.
- py2nim → Nim parser: pipeline · Nim driver runs the shared parse-to-Nim flow.
- py2nim → Nim stdlib: runtime support · Nim builds rely on the local support library.
- Benchmarks → py2py: exercise · Benchmark corpus validates the Python backend and parser.
- Benchmarks → py2nim: exercise · Benchmark corpus validates the Nim backend and build path.
- Nim tests → py2nim: regress · Nim-focused tests cover backend lowering and imports/types.
- Py tests → py2py: regress · Python-focused tests cover backend output behavior.