How lemon-lang/lemon works
Project type and purpose
• This is a language implementation project: the Lemon Programming Language. It contains both a compiler and a virtual machine (VM), along with core libraries for the language.
• Key indicators:
• src/: source code for compiler front-end, code generator, VM (“machine”), memory management, and REPL shell
• lib/: built-in libraries (OS and socket support)
• test/: sample programs to validate compilation and execution
File structure
• Top-level:
• src/ – compiler and VM implementation in C
• lib/ – core runtime libraries for OS and socket modules
• doc/ – documentation of source code and design rationale
• test/ – lemon source files for unit testing
• Makefile, VERSION, LICENSE, README.md
• Patterns and architectural choices:
• Compiler broken into lexer (lexer.c/token.c), parser (parser.c/syntax.c), semantic scope (scope.c/symbol.c), AST manipulation, peephole optimizer, code generator (generator.c), and output of bytecode (opcode.c)
• VM separated into memory allocator (allocator.c, arena.c, mpool.c), garbage collector (collector.c), interpreter loop (machine.c), and built-in object implementations (lstring.c, larray.c, etc.)
• Shell (shell.c) as REPL front-end invoking compiler + VM
README for additional insights
• README clarifies build flags controlling debug, static linking, malloc wrapper, and enabling OS/socket modules.
a. Main components
1. Front-end Compiler
– Lexer & Tokenizer
– Parser & AST builder
– Semantic Analyzer & Scope Manager
– Intermediate Representation & Peephole Optimizer
– Bytecode Generator (generator.c + opcode definitions)
2. Virtual Machine
– Memory Management (allocator, arena, mpool)
– Garbage Collector (collector)
– Interpreter Loop (machine)
– Built-in Object Implementations (lstring, larray, etc.)
3. Core Libraries (lib/)
– OS Module (lib/os.c, lib/os.h)
– Socket Module (lib/socket.c, lib/socket.h)
4. REPL Shell (shell.c)
5. Test Harness (test/*.lm)
6. Build System (Makefile)
b. Relationships and interactions
– Source File (.lm) enters REPL Shell or CLI driver (main.c)
– Driver invokes Compiler Front-end
→ passes code through Lexer → Parser → Semantic Analyzer → IR → Bytecode Generator
– Generated Bytecode passed into VM Interpreter
→ VM uses Memory Manager + GC to allocate objects
→ VM executes bytecode instructions, calling built-in implementations or external libs
– OS/Socket modules are dynamically linked (or statically if STATIC=1) and invoked via VM built-ins
– Test harness automates invocation of compiler+VM and verifies output
c. Architectural patterns and design principles
– Modular separation: front-end vs VM vs built-ins
– Layered architecture:
1. Presentation (shell/main driver)
2. Compilation pipeline
3. Execution engine
4. Runtime libraries
– Pluggable modules: OS and socket support toggled at build time
– Memory safety via custom allocator and garbage collector
d. Key technologies/frameworks/libraries
– ANSI C99 for all components
– Makefile for build orchestration
– No external dependencies beyond standard C library and POSIX/BSD socket APIs