How arthkamboj/compiler-lisp-mvp works
1. Overall shape
This repo is a small Lisp implementation with a classic front-end / runtime split: source text is scanned, parsed, compiled to bytecode, then executed by a custom virtual machine with custom memory management.
2. Entry point and execution modes
main.cpp is the orchestrator. It supports two top-level modes: an interactive REPL that preserves state across inputs, and batch execution of a .lisp script passed on the command line. It also appears to bootstrap a self-hosted standard library at startup.
3. Source representation and value model
types.h defines the language’s core data model: Lisp values, opcode definitions, and the variant-based runtime type system. This file is the shared contract used across lexer, parser, compiler, VM, environment, and GC.
4. Lexing boundary
lexer.cpp/.h convert raw text into tokens. The lexer is a handwritten component rather than a generated one, so it is the first boundary between unstructured input and language structure.
5. Parsing boundary
parser.cpp/.h take the token stream and build the AST. The repository uses recursive descent parsing, which means syntax forms like function definitions, conditionals, loops, and nested expressions are represented as tree nodes before compilation.
6. Compilation boundary
compiler.cpp/.h lower the AST into linear bytecode. This is where high-level Lisp forms become stack-machine operations, including arithmetic, variable access, function calls, and control-flow jumps. if and while rely on backpatching jump offsets after the target locations are known.
7. Runtime execution
vm.cpp/.h implement a stack-based virtual machine. The VM executes the bytecode with an instruction pointer, value stack, and call-frame stack. This is the core execution engine for both user code and the bootstrapped standard library.
8. Lexical scoping and closures
environment.h defines chained variable environments for nested scopes. It is the mechanism that makes define, local bindings, and closures work correctly: functions can capture their birth environment and later resolve referenced variables through that chain.
9. Garbage collection
gc.cpp/.h provide a custom mark-and-sweep collector. Memory management is not delegated to C++ smart pointers; instead, the GC traces live roots from VM state and environments, then reclaims unreachable heap objects such as lists and captured environments.
10. Primary data flow
The main pipeline is:
raw Lisp source → lexer tokens → AST → bytecode → VM execution → heap-allocated Lisp values.
Control flow and function calls remain inside the bytecode/VM layer, while scope resolution and captured closures bridge the compiler and runtime.
11. Bootstrapped standard library
The README indicates the standard library is written in Micro-Lisp itself and loaded dynamically on boot. Architecturally, that means the VM must be functional enough to run library code before user input is evaluated, and the library becomes part of the initial runtime state rather than a separate native module.
12. Notable architectural boundaries
The repo has a clear separation between language front end, execution engine, and memory manager:
- front end: lexer + parser + compiler
- runtime: VM + environments
- memory: GC
- orchestration: main.cpp
This makes it straightforward to diagram as a pipeline with shared runtime state around the VM and GC.
Main entry points: main.cpp main.cpp · vm.cpp VM · types.h types.h · gc.cpp GC
Components 14
Orchestration
Startup and execution entry points
- main.cpp · Coordinates startup, REPL, script execution, and stdlib bootstrap
src/main.cpp - Stdlib boot · Self-hosted standard library loaded at startup
Front End
Text-to-bytecode compilation pipeline
- Lexer · Handwritten source tokenizer
src/lexer.cpp - Tokens · Lexical units passed from lexer to parser
- Parser · Builds AST from token stream
src/parser.cpp - AST · Structured Lisp forms before compilation
- Compiler · Lowers AST to stack-machine bytecode and backpatches control flow
src/compiler.cpp - Bytecode · Linear instructions executed by the VM
- types.h · Core value, opcode, and runtime type definitions shared across the system
src/types.h
Runtime
Execution engine and scope state
- VM · Executes bytecode with stack and call frames
src/vm.cpp - Environment · Nested bindings for lexical scope and closures
src/environment.h - Values · Heap-visible runtime objects manipulated by the VM
Memory
Heap management and collection
- GC · Traces live roots from VM state and environments
src/gc.cpp - Heap · Allocated objects reclaimed by the collector
Connections 23
- main.cpp → Stdlib boot: boots · Loads the self-hosted standard library during startup
- main.cpp → Lexer: drives · Routes input through the language pipeline
- main.cpp → Parser: drives · Coordinates parsing of source input
- main.cpp → Compiler: drives · Invokes compilation for executable forms
- main.cpp → VM: runs · Executes compiled programs and REPL input
- main.cpp → GC: manages · Triggers or relies on collection during execution
- Lexer → Tokens: produces · Converts raw source text into tokens
- Tokens → Parser: feeds · Parser consumes the token stream
- Parser → AST: produces · Builds structured syntax trees
- AST → Compiler: feeds · AST is lowered into bytecode
- Compiler → Bytecode: produces · Emits linear instruction sequences
- Bytecode → VM: executes · VM interprets the compiled instructions
- VM → Values: operates on · Evaluates and returns runtime Lisp values
- VM → Environment: resolves · Uses chained environments for lexical scope
- Environment → Values: binds · Stores variable bindings to runtime values
- GC → VM: roots from · Uses VM state as a root set for tracing
- GC → Environment: roots from · Traverses live environments and captured closures
- GC → Heap: reclaims · Collects unreachable heap-allocated objects
- types.h → Lexer: shared contract · Lexical and token definitions are shared types
- types.h → Parser: shared contract · Parser uses shared value and syntax definitions
- types.h → Compiler: shared contract · Compiler emits typed opcodes and values
- types.h → VM: shared contract · VM executes the common opcode/value model
- types.h → GC: shared contract · Collector understands shared heap object layouts