How eldarsalmanow/gslanguage works
1. What this repo is
GSLanguage is a C++20 implementation of a compiled programming language. The repository is organized as a compiler toolchain rather than an application: it takes source text through analysis stages and ends with code generation.
2. Top-level architecture
The project is split into clear compiler subsystems under include/ and src/: IO, Lexer, Parser, AST, Semantic, Optimizer, CodeGenerator, Driver, and Debug. The mirrored include/... and src/... layout indicates a library-style core with headers exposing the compiler API and source files implementing each stage.
3. Input and source management
The IO layer is the entry point for files/streams/messages. Classes like GS_Source, GS_Reader, and GS_Stream likely normalize input from files or buffers, while GS_Message provides structured diagnostics. This layer forms the boundary between raw text and compiler processing.
4. Lexing and tokenization
The Lexer subsystem converts source text into tokens. It includes cursor/token abstractions and token type definitions in GS_TokenTypes.def, suggesting a centralized token catalog used by both lexer and parser. This stage produces the stream consumed by parsing.
5. Parsing and grammar
The Parser turns tokens into the language’s AST. The presence of GRAMMAR.md and GS_Parser indicates the grammar is documented separately from implementation. Parsing is the syntactic boundary: it translates language structure into tree nodes without doing type or semantic resolution.
6. AST as the central internal model
The AST directory is the shared core model used by downstream stages. It contains base node types plus concrete declarations, statements, and expressions such as functions, imports, modules, literals, binary/unary operations, control flow, calls, arrays, ranges, and indexing. GS_ASTBuilder and GS_ASTContext suggest AST construction is context-aware and likely owns node creation and source association.
7. Semantic analysis and symbol/type state
The Semantic subsystem resolves meaning after parsing. It includes GS_TableOfSymbols, GS_Type, and semantic passes such as GS_SymbolsPlaceholderPass and GS_TypeInferencePass. This stage is where identifiers, scopes, and inferred types are established, forming the semantic boundary before optimization or emission.
8. Optimization
The Optimizer currently appears small and pass-based, with GS_ConstantFoldingPass as the notable transformation. It likely operates on the analyzed AST or an intermediate representation derived from it, performing local compile-time simplifications before code generation.
9. Code generation and backend abstraction
CodeGenerator provides an abstraction over emission targets via GS_Backend and GS_CodeGenerator, with an LLVM-specific implementation in CodeGenerator/LLVM. GS_LLVMVisitor and GS_LLVMBackend show that LLVM is a material external dependency and the current backend target. The architecture is backend-oriented, so the compiler can be extended with additional targets behind the same interface.
10. Driver, sessions, and orchestration
The Driver layer coordinates the compilation pipeline. Classes like GS_Compiler, GS_SessionManager, GS_Session, GS_GlobalContext, GS_Arguments, and GS_QueryManager suggest a higher-level API for running compilations, managing state across runs, and handling queries or commands. This is the control plane that ties IO, parsing, semantics, optimization, and codegen together.
11. Debugging and observability
Debug contains AST and symbol-table dumpers. These are important architectural tools because they expose internal compiler state at stage boundaries, making it easier to inspect how source text becomes AST, symbols, and types.
12. Examples, tests, and build system
The repository uses CMake throughout, with an external/conanfile.txt indicating dependency management through Conan. examples/ includes sample .gs, .c, .rs, and assembly files, likely used to exercise compilation or integration points. tests/ currently focuses on IO, Lexer, and Parser, which matches the earliest pipeline stages where correctness is most foundational.
Main entry points: Driver Driver · Parser Parser · Semantic Semantic · IO IO
Components 18
Compiler core
Main compilation pipeline from source input to generated code.
- IO · Reads sources, streams, and compiler messages.
include/IO - Lexer · Converts source text into tokens.
include/Lexer - Parser · Builds syntax trees from the token stream.
include/Parser - AST · Shared tree representation used by later compiler stages.
include/AST - Semantic · Resolves symbols, scopes, and types.
include/Semantic - Optimizer · Performs compile-time simplifications on the analyzed tree.
include/Optimizer - CodeGen · Emits target code through a backend abstraction.
include/CodeGenerator - LLVM · LLVM-backed code generation implementation.
include/CodeGenerator/LLVM
Public API
Headers exposing the compiler subsystems.
- IO API · Public IO interfaces and message types.
include/IO/IO.h - Lexer API · Public lexer interfaces and token definitions.
include/Lexer/Lexer.h - Parser API · Public parser interface and grammar documentation.
include/Parser/Parser.h - AST API · Public AST node, visitor, and builder interfaces.
include/AST/AST.h - Semantic API · Public semantic analysis types and passes.
include/Semantic/Semantic.h - CodeGen API · Public backend and code generator interfaces.
include/CodeGenerator/CodeGenerator.h
Support
Orchestration, diagnostics, and examples around the core compiler.
- Driver · Coordinates sessions, arguments, queries, and compilation runs.
include/Driver - Debug · AST and symbol-table dumpers for observing compiler state.
include/Debug - Examples · Sample inputs and integration examples for the compiler.
examples - Tests · Unit tests for IO, lexer, and parser behavior.
tests
Connections 24
- IO → Lexer: source text · Normalized source input flows into lexical analysis.
- Lexer → Parser: tokens · Token stream feeds parsing.
- Parser → AST: trees · Parsing produces the AST as the central internal model.
- AST → Semantic: analysis input · Semantic passes operate on AST nodes.
- Semantic → Optimizer: typed tree · Optimization runs after meaning is established.
- Optimizer → CodeGen: lowered tree · Optimized representation is emitted by code generation.
- CodeGen → LLVM: backend impl · LLVM is the concrete target backend.
- Driver → IO: orchestrates · Driver configures source and diagnostic handling.
- Driver → Parser: runs pipeline · Compiler orchestration invokes parsing and later stages.
- Driver → Semantic: runs pipeline · Driver controls semantic analysis.
- Driver → Optimizer: runs pipeline · Driver controls optimization passes.
- Driver → CodeGen: runs pipeline · Driver triggers backend emission.
- Debug → AST: dumps · AST dumper inspects tree structure.
- Debug → Semantic: dumps · Symbol-table dumper inspects semantic state.
- Examples → Driver: exercise · Sample programs demonstrate compiler usage.
- Tests → IO: covers · IO tests validate the input layer.
- Tests → Lexer: covers · Lexer tests validate tokenization.
- Tests → Parser: covers · Parser tests validate grammar handling.
- IO API → IO: exposes · Public IO headers define the IO layer API.
- Lexer API → Lexer: exposes · Public lexer headers define tokenization APIs.
- Parser API → Parser: exposes · Public parser headers define syntax analysis APIs.
- AST API → AST: exposes · Public AST headers define the shared tree model.
- Semantic API → Semantic: exposes · Public semantic headers define analysis APIs.
- CodeGen API → CodeGen: exposes · Public code generation headers define backend APIs.