How happyseaweed/lemon-language works
1. What this repo is
- This is a work-in-progress compiler for the Lemon language.
- The language is designed around basic control flow, functions, and tensor-oriented operations, with LLVM used for IR generation, optimization, and execution.
2. Top-level structure
- The repository contains two compiler generations: lemon-0 and lemon-1.
- lemon-0 is the smaller initial pipeline.
- lemon-1 is the more complete architecture and is the best reference for the current system design.
3. Source inputs and examples
- .lem files are Lemon programs.
- The repo includes sample programs such as example.lem, simple.lem, complex.lem, tensor.lem, and test/sanity.lem.
- grammar.md in lemon-1 documents the language syntax, so it acts as the contract between the parser and the rest of the compiler.
4. Front-end pipeline
- The compiler pipeline starts with Lexer, which tokenizes source text.
- Parser consumes tokens and builds the AST.
- AST is the central internal representation shared by later phases.
- In lemon-0, this front-end is the main focus; in lemon-1, it feeds additional analysis and lowering stages.
5. Semantic and type layer in lemon-1
- SemanticAnalyzer validates the parsed program before code generation.
- SymbolTable tracks scoped identifiers, functions, and variables.
- Types defines the language’s type system, including the key distinction between float and tensor.
- BuilderStack appears to support nested construction or traversal state during analysis/codegen, likely helping manage scoped blocks and expression building.
6. Tensor-specific runtime model
- Tensor is a dedicated subsystem in lemon-1, indicating tensors are not just syntactic sugar but a first-class runtime/compiler concept.
- This likely forms the bridge between high-level tensor syntax and the generated LLVM-side representation.
- The presence of tensor-specific files suggests the compiler is intentionally structured around ML-style data operations rather than only scalar arithmetic.
7. Code generation and LLVM boundary
- Codegen is the lowering boundary from validated AST into LLVM IR.
- output.ll at the repo root suggests the compiler can emit LLVM IR as a visible artifact.
- This separation means the compiler architecture is: parse and analyze in Lemon-specific code, then hand off to LLVM for optimization/execution.
8. Execution/JIT boundary
- LemonJIT.h exists in both versions, showing the project supports JIT-style execution rather than only static compilation.
- The JIT sits at the end of the pipeline after IR generation, acting as the runtime execution layer for compiled Lemon code.
9. Entry points and tooling
- src/lemon.cc is the executable entry point in both versions.
- CMake is the build system, so the repo is organized around a native C++ build and binary output.
- This is not an interpreted scripting project; it is a compiled toolchain with a command-line driver.
10. Debugging and observability
- ShowAST.cc in lemon-1 is a diagnostic/inspection utility for visualizing or printing the AST.
- That indicates the compiler has explicit developer-facing tooling to inspect internal IR, which is useful for architecture separation and debugging parser/semantic issues.
11. Version progression
- lemon-0 contains the basic compiler skeleton: lexer, parser, AST, and JIT integration.
- lemon-1 expands that into a real language implementation with semantic analysis, symbol management, tensor support, type definitions, and LLVM code generation.
- For architecture diagrams, lemon-1 should be treated as the authoritative pipeline, with lemon-0 as an earlier prototype.
12. Overall data flow
- The main flow is: .lem source file → lexer → parser → AST → semantic analysis / symbol resolution / type handling → code generation to LLVM IR → JIT execution or IR output.
- The main architectural boundary is between Lemon-specific language processing and LLVM-backed lowering/execution.
- The key domain-specific subsystem is tensor support, which likely influences parsing, typing, and codegen more than scalar-only features would.
Main entry points: SemanticAnalyzer.h Semantic · AST.h AST · Codegen.h Codegen · lemon.cc Driver
Components 21
Repository
Top-level Lemon compiler workspace with two generations and shared examples.
lemon-0
Early compiler prototype focused on lexing, parsing, AST, and JIT hookup.
- Driver · Command-line entry point for the first compiler generation.
lemon-0/src/lemon.cc - Lexer · Tokenizes Lemon source text.
lemon-0/include/Lexer.h - Parser · Builds the AST from token streams.
lemon-0/include/Parser.h - AST · Shared abstract syntax tree used by later phases.
lemon-0/include/AST.h - JIT · Execution boundary for compiled code in the prototype pipeline.
lemon-0/include/LemonJIT.h
lemon-1
Current compiler architecture with semantics, tensors, lowering, and diagnostics.
- Driver · Command-line entry point for the current compiler generation.
lemon-1/src/lemon.cc - Lexer · Tokenizes Lemon source text.
lemon-1/include/Lexer.h - Parser · Constructs the AST from tokens.
lemon-1/include/Parser.h - AST · Central internal representation shared across analysis and lowering.
lemon-1/include/AST.h - Semantic · Validates names, scopes, and program rules before code generation.
lemon-1/include/SemanticAnalyzer.h - Symbols · Tracks scoped identifiers, functions, and variables.
lemon-1/include/SymbolTable.h - Types · Defines Lemon types, including scalar and tensor distinctions.
lemon-1/include/Types.h - Builder Stack · Supports nested construction and traversal state during compilation.
lemon-1/include/BuilderStack.h - Tensor · First-class tensor representation and lowering support.
lemon-1/include/Tensor.h - Codegen · Lowers validated Lemon AST into LLVM IR.
lemon-1/include/Codegen.h - JIT · LLVM-backed execution boundary for compiled Lemon programs.
lemon-1/include/LemonJIT.h - Show AST · Developer utility for inspecting the AST.
lemon-1/src/ShowAST.cc
Connections 23
- README → Driver: describes · Repository documentation points to the prototype compiler.
- README → Driver: describes · Repository documentation points to the current compiler.
- Example → Driver: feeds · Sample Lemon source can be compiled by the prototype.
- Example → Driver: feeds · Sample Lemon source can be compiled by the current pipeline.
- Grammar → Parser: defines · Grammar is the contract consumed by the parser.
- Driver → Lexer: starts · Driver invokes lexical analysis.
- Lexer → Parser: tokens · Tokens are passed to the parser.
- Parser → AST: builds · Parser produces the AST.
- AST → JIT: executes via · Prototype pipeline hands off to JIT execution.
- Driver → Lexer: starts · Driver invokes lexical analysis.
- Lexer → Parser: tokens · Tokens are passed to the parser.
- Parser → AST: builds · Parser produces the AST.
- AST → Semantic: analyzed by · Semantic analysis validates the AST.
- Semantic → Symbols: uses · Semantic analysis relies on scoped symbol tracking.
- Semantic → Types: checks · Semantic analysis enforces type rules.
- Semantic → Builder Stack: tracks · Analysis uses builder state for nested traversal/construction.
- AST → Tensor: contains · Tensor constructs are part of the language model.
- Tensor → Types: typed by · Tensor handling depends on the type system.
- Semantic → Codegen: lowers to · Validated programs are lowered for code generation.
- Codegen → LLVM IR: emits · Code generation produces LLVM IR output.
- Codegen → JIT: hands off to · Lowered IR can be executed by the JIT boundary.
- AST → Show AST: inspected by · AST inspection utility reads the tree for debugging.
- JIT → JIT: evolves into · The runtime boundary persists across compiler generations.