# elkarouh/hparsec architecture

> Architecture diagram of the GitHub repository elkarouh/hparsec, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/elkarouh/hparsec
- Repository: https://github.com/elkarouh/hparsec

## Overview

##### What this repository is
Adascript is a source-to-source compiler for a Python-like language. The same `.ady` input is parsed once and then emitted into two target runtimes: **Python 3** and **Nim**. The repo is organized around that split-brain backend design rather than around an application domain.

##### Top-level pipeline
The main data flow is: Adascript source → tokenizer → parser/grammar → AST-like node objects → backend-specific code generation. The two entrypoints, `TO_PYTHON/py2py.py` and `TO_NIM/py2nim.py`, drive that pipeline and differ only in the final emission target and some Nim-specific build orchestration.

##### Lexing and source normalization
`hek_tokenize.py` is the front door for source text. It does more than tokenization: it preprocesses Ada-style tick attributes like `Type'Attr`, preserves comment/newline structure with `RichNL`, and normalizes newline handling inside brackets. `hek_helpers.py` provides shared indentation and newline utilities used across the parser and emitters.

##### Parser combinator core
`hek_parsec.py` is the parser engine underneath the grammar. It provides combinator-style parsing primitives, packrat memoization, forward references, and parser-state management. This is the foundation that lets the language grammar be expressed declaratively instead of with hand-written recursive descent code.

##### Language grammar definition
`ADASCRIPT_GRAMMAR/` contains the language-neutral grammar split by concern: expressions, simple statements, compound statements, and declarations. These modules define the syntax of Adascript itself, including Python compatibility plus additive features such as type declarations, ranges, `case/when`, shell statements, and shell variables.

##### AST-to-target transformation boundary
The grammar nodes are not the final output; each node type gets target-specific `to_py()` or `to_nim()` methods attached in the backend packages. This is the key architectural boundary: the same parsed structure is lowered differently depending on whether the output is Python or Nim. The backend modules mirror the grammar split with expression, statement, parser, and declaration emitters.

##### Python backend
`TO_PYTHON/` translates Adascript into valid Python 3 source. It maps Adascript types to Python typing constructs, expands enums/dataclasses/variant records into Python class forms, rewrites shell and bash-style features into Python stdlib calls, and preserves Python-native semantics where possible. The backend is designed to stay close to idiomatic Python output, not just syntactic equivalence.

##### Nim backend
`TO_NIM/` performs the analogous lowering into Nim. It adds Nim-specific constructs such as enum-indexed arrays, objects vs. ref objects, Nim `case/of`, and Nim runtime/import mapping. `py2nim.py` also contains build/runtime orchestration beyond transpilation, including incremental rebuild checks, cache management, and optional compile-and-run behavior.

##### Runtime and interoperability support
The generated code relies on different runtime assumptions in each target. Python output leans on the standard library and typing/dataclasses; Nim output sometimes needs extra support from a local `BENCHMARK/stdlib.nim` shim for Python-like conveniences such as queues and helper abstractions. The repo also handles Python-to-Nim interop through `nimpy` when a Python library has no direct Nim equivalent.

Main entry points: [`TO_PYTHON/hek_py3_parser.py`](https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py3_parser.py) (Py parser), [`TO_NIM/hek_nim_parser.py`](https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_parser.py) (Nim parser), [`hek_parsec.py`](https://github.com/elkarouh/hparsec/blob/master/hek_parsec.py) (Parsec core), [`TO_NIM/py2nim.py`](https://github.com/elkarouh/hparsec/blob/master/TO_NIM/py2nim.py) (py2nim)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_frontend["Frontend"]
  node_n_tokenize["Tokenizer<br/>source normalizer<br/>[hek_tokenize.py]"]
  node_n_helpers["Helpers<br/>shared utils<br/>[hek_helpers.py]"]
  node_n_exprg["Expr grammar<br/>grammar module<br/>[py3expr.py]"]
  node_n_stmtg["Stmt grammar<br/>grammar module<br/>[py3stmt.py]"]
  node_n_compg["Compound grammar<br/>grammar module"]
  node_n_declg["Decl grammar<br/>grammar module<br/>[py_declarations.py]"]
end

subgraph group_g_parsecore["Parse core"]
  node_n_parsec["Parsec core<br/>parser engine<br/>[hek_parsec.py]"]
end

subgraph group_g_backends["Backends"]
  node_n_py_parser["Py parser<br/>python backend parser<br/>[hek_py3_parser.py]"]
  node_n_py_expr["Py expr<br/>python emitter<br/>[hek_py3_expr.py]"]
  node_n_py_stmt["Py stmt<br/>python emitter<br/>[hek_py3_stmt.py]"]
  node_n_py_decl["Py decl<br/>python emitter"]
  node_n_py_driver["py2py<br/>python transpiler<br/>[py2py.py]"]
  node_n_nim_parser["Nim parser<br/>nim backend parser<br/>[hek_nim_parser.py]"]
  node_n_nim_expr["Nim expr<br/>nim emitter<br/>[hek_nim_expr.py]"]
  node_n_nim_stmt["Nim stmt<br/>nim emitter<br/>[hek_nim_stmt.py]"]
  node_n_nim_decl["Nim decl<br/>nim emitter"]
  node_n_nim_driver["py2nim<br/>nim transpiler/build driver<br/>[py2nim.py]"]
  node_n_nim_stdlib["Nim stdlib<br/>runtime shim<br/>[stdlib.nim]"]
end

subgraph group_g_validation["Validation"]
  node_n_bench["Benchmarks<br/>integration corpus"]
  node_n_nim_tests["Nim tests<br/>regression suite"]
  node_n_py_tests["Py tests<br/>regression suite<br/>[test.py]"]
end

node_n_tokenize -->|"tokens"| node_n_parsec
node_n_helpers -->|"newline/indent"| node_n_tokenize
node_n_helpers -->|"formatting"| node_n_py_stmt
node_n_helpers -->|"formatting"| node_n_nim_stmt
node_n_parsec -->|"drives"| node_n_exprg
node_n_parsec -->|"drives"| node_n_stmtg
node_n_parsec -->|"drives"| node_n_compg
node_n_parsec -->|"drives"| node_n_declg
node_n_exprg -->|"AST nodes"| node_n_py_parser
node_n_stmtg -->|"AST nodes"| node_n_py_parser
node_n_compg -->|"AST nodes"| node_n_py_parser
node_n_declg -->|"AST nodes"| node_n_py_parser
node_n_exprg -->|"AST nodes"| node_n_nim_parser
node_n_stmtg -->|"AST nodes"| node_n_nim_parser
node_n_compg -->|"AST nodes"| node_n_nim_parser
node_n_declg -->|"AST nodes"| node_n_nim_parser
node_n_py_parser -->|"emits"| node_n_py_expr
node_n_py_parser -->|"emits"| node_n_py_stmt
node_n_py_parser -->|"emits"| node_n_py_decl
node_n_nim_parser -->|"emits"| node_n_nim_expr
node_n_nim_parser -->|"emits"| node_n_nim_stmt
node_n_nim_parser -->|"emits"| node_n_nim_decl
node_n_py_driver -->|"pipeline"| node_n_py_parser
node_n_nim_driver -->|"pipeline"| node_n_nim_parser
node_n_nim_driver -->|"runtime support"| node_n_nim_stdlib
node_n_bench -->|"exercise"| node_n_py_driver
node_n_bench -->|"exercise"| node_n_nim_driver
node_n_nim_tests -->|"regress"| node_n_nim_driver
node_n_py_tests -->|"regress"| node_n_py_driver

click node_n_tokenize "https://github.com/elkarouh/hparsec/blob/master/hek_tokenize.py"
click node_n_helpers "https://github.com/elkarouh/hparsec/blob/master/hek_helpers.py"
click node_n_parsec "https://github.com/elkarouh/hparsec/blob/master/hek_parsec.py"
click node_n_exprg "https://github.com/elkarouh/hparsec/blob/master/ADASCRIPT_GRAMMAR/py3expr.py"
click node_n_stmtg "https://github.com/elkarouh/hparsec/blob/master/ADASCRIPT_GRAMMAR/py3stmt.py"
click node_n_compg "https://github.com/elkarouh/hparsec/blob/master/ADASCRIPT_GRAMMAR/py3compound_stmt.py"
click node_n_declg "https://github.com/elkarouh/hparsec/blob/master/ADASCRIPT_GRAMMAR/py_declarations.py"
click node_n_py_parser "https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py3_parser.py"
click node_n_py_expr "https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py3_expr.py"
click node_n_py_stmt "https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py3_stmt.py"
click node_n_py_decl "https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py_declarations.py"
click node_n_py_driver "https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/py2py.py"
click node_n_nim_parser "https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_parser.py"
click node_n_nim_expr "https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_expr.py"
click node_n_nim_stmt "https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_stmt.py"
click node_n_nim_decl "https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_declarations.py"
click node_n_nim_driver "https://github.com/elkarouh/hparsec/blob/master/TO_NIM/py2nim.py"
click node_n_nim_stdlib "https://github.com/elkarouh/hparsec/blob/master/TO_NIM/stdlib.nim"
click node_n_bench "https://github.com/elkarouh/hparsec/tree/master/BENCHMARK"
click node_n_nim_tests "https://github.com/elkarouh/hparsec/tree/master/TO_NIM/TEST"
click node_n_py_tests "https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/test.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_tokenize,node_n_helpers,node_n_exprg,node_n_stmtg,node_n_compg,node_n_declg toneBlue
class node_n_parsec toneAmber
class node_n_py_parser,node_n_py_expr,node_n_py_stmt,node_n_py_decl,node_n_py_driver,node_n_nim_parser,node_n_nim_expr,node_n_nim_stmt,node_n_nim_decl,node_n_nim_driver,node_n_nim_stdlib toneMint
class node_n_bench,node_n_nim_tests,node_n_py_tests toneRose
```

## Components

### Frontend

Source handling and language grammar

- [Tokenizer](https://github.com/elkarouh/hparsec/blob/master/hek_tokenize.py): source normalizer · Front door for .ady text; handles tokenization, tick attributes, rich newlines, and bracket-aware normalization. · `hek_tokenize.py`
- [Helpers](https://github.com/elkarouh/hparsec/blob/master/hek_helpers.py): shared utils · Indentation and newline helpers used across parsing and emission. · `hek_helpers.py`
- [Expr grammar](https://github.com/elkarouh/hparsec/blob/master/ADASCRIPT_GRAMMAR/py3expr.py): grammar module · Expression syntax and precedence rules for Adascript. · `ADASCRIPT_GRAMMAR/py3expr.py`
- [Stmt grammar](https://github.com/elkarouh/hparsec/blob/master/ADASCRIPT_GRAMMAR/py3stmt.py): grammar module · Simple statement syntax for the shared language grammar. · `ADASCRIPT_GRAMMAR/py3stmt.py`
- [Compound grammar](https://github.com/elkarouh/hparsec/blob/master/ADASCRIPT_GRAMMAR/py3compound_stmt.py): grammar module · Compound statements, control flow, and block forms. · `ADASCRIPT_GRAMMAR/py3compound_stmt.py`
- [Decl grammar](https://github.com/elkarouh/hparsec/blob/master/ADASCRIPT_GRAMMAR/py_declarations.py): grammar module · Declarations, type forms, and other top-level language constructs. · `ADASCRIPT_GRAMMAR/py_declarations.py`

### Parse core

Shared parser machinery

- [Parsec core](https://github.com/elkarouh/hparsec/blob/master/hek_parsec.py): parser engine · Combinator parser framework with memoization, forward refs, and parser state. · `hek_parsec.py`

### Backends

Target-specific lowering and build drivers

- [Py parser](https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py3_parser.py): python backend parser · Python-target parser glue that attaches emission methods to grammar nodes. · `TO_PYTHON/hek_py3_parser.py`
- [Py expr](https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py3_expr.py): python emitter · Expression lowering for Python 3 output. · `TO_PYTHON/hek_py3_expr.py`
- [Py stmt](https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py3_stmt.py): python emitter · Statement lowering for Python 3 output. · `TO_PYTHON/hek_py3_stmt.py`
- [Py decl](https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/hek_py_declarations.py): python emitter · Declaration and type lowering for Python output. · `TO_PYTHON/hek_py_declarations.py`
- [py2py](https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/py2py.py): python transpiler · Command-line driver that runs the parse-to-Python pipeline. · `TO_PYTHON/py2py.py`
- [Nim parser](https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_parser.py): nim backend parser · Nim-target parser glue that connects grammar nodes to Nim emission. · `TO_NIM/hek_nim_parser.py`
- [Nim expr](https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_expr.py): nim emitter · Expression lowering for Nim output. · `TO_NIM/hek_nim_expr.py`
- [Nim stmt](https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_stmt.py): nim emitter · Statement lowering for Nim output. · `TO_NIM/hek_nim_stmt.py`
- [Nim decl](https://github.com/elkarouh/hparsec/blob/master/TO_NIM/hek_nim_declarations.py): nim emitter · Declaration and type lowering for Nim output. · `TO_NIM/hek_nim_declarations.py`
- [py2nim](https://github.com/elkarouh/hparsec/blob/master/TO_NIM/py2nim.py): nim transpiler/build driver · Command-line driver plus incremental build, cache, and optional compile-and-run orchestration. · `TO_NIM/py2nim.py`
- [Nim stdlib](https://github.com/elkarouh/hparsec/blob/master/TO_NIM/stdlib.nim): runtime shim · Local Nim support library for Python-like conveniences and helper abstractions. · `TO_NIM/stdlib.nim`

### Validation

Benchmarks and regression coverage

- [Benchmarks](https://github.com/elkarouh/hparsec/tree/master/BENCHMARK): integration corpus · Adascript programs and reference outputs that exercise major language features and workloads. · `BENCHMARK`
- [Nim tests](https://github.com/elkarouh/hparsec/tree/master/TO_NIM/TEST): regression suite · Focused backend and import/type regression cases for Nim output. · `TO_NIM/TEST`
- [Py tests](https://github.com/elkarouh/hparsec/blob/master/TO_PYTHON/test.py): regression suite · Focused Python backend regression checks. · `TO_PYTHON/test.py`

## Connections

- Tokenizer → Parsec core: tokens · Source text is normalized and tokenized before parsing.
- Helpers → Tokenizer: newline/indent · Shared formatting utilities support tokenization and parsing.
- Helpers → Py stmt: formatting · Python emission uses shared indentation and newline helpers.
- Helpers → Nim stmt: formatting · Nim emission uses shared indentation and newline helpers.
- Parsec core → Expr grammar: drives · Parser combinators are used to express expression grammar.
- Parsec core → Stmt grammar: drives · Parser combinators are used to express statement grammar.
- Parsec core → Compound grammar: drives · Parser combinators are used to express compound statement grammar.
- Parsec core → Decl grammar: drives · Parser combinators are used to express declaration grammar.
- Expr grammar → Py parser: AST nodes · Grammar nodes are extended with Python emission behavior.
- Stmt grammar → Py parser: AST nodes · Grammar nodes are extended with Python emission behavior.
- Compound grammar → Py parser: AST nodes · Grammar nodes are extended with Python emission behavior.
- Decl grammar → Py parser: AST nodes · Grammar nodes are extended with Python emission behavior.
- Expr grammar → Nim parser: AST nodes · Grammar nodes are extended with Nim emission behavior.
- Stmt grammar → Nim parser: AST nodes · Grammar nodes are extended with Nim emission behavior.
- Compound grammar → Nim parser: AST nodes · Grammar nodes are extended with Nim emission behavior.
- Decl grammar → Nim parser: AST nodes · Grammar nodes are extended with Nim emission behavior.
- Py parser → Py expr: emits · Python parser glue dispatches to expression lowering.
- Py parser → Py stmt: emits · Python parser glue dispatches to statement lowering.
- Py parser → Py decl: emits · Python parser glue dispatches to declaration lowering.
- Nim parser → Nim expr: emits · Nim parser glue dispatches to expression lowering.
- Nim parser → Nim stmt: emits · Nim parser glue dispatches to statement lowering.
- Nim parser → Nim decl: emits · Nim parser glue dispatches to declaration lowering.
- py2py → Py parser: pipeline · Python driver runs the shared parse-to-Python flow.
- py2nim → Nim parser: pipeline · Nim driver runs the shared parse-to-Nim flow.
- py2nim → Nim stdlib: runtime support · Nim builds rely on the local support library.
- Benchmarks → py2py: exercise · Benchmark corpus validates the Python backend and parser.
- Benchmarks → py2nim: exercise · Benchmark corpus validates the Nim backend and build path.
- Nim tests → py2nim: regress · Nim-focused tests cover backend lowering and imports/types.
- Py tests → py2py: regress · Python-focused tests cover backend output behavior.
