# github/semantic architecture

> Architecture diagram of the GitHub repository github/semantic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/github/semantic
- Repository: https://github.com/github/semantic

## Overview

This project is a multi-language static analysis tool written in Haskell. Its primary purpose is to parse, analyze, and compare source code across a wide variety of languages such as Ruby, JavaScript/TypeScript, Python, Go, PHP, and many others. The tool uses external grammars from tree‑sitter to generate language-specific syntax types, reads “blobs” (chunks of source code) either from the filesystem or via protocol buffer requests, and then applies analysis algorithms (including diff algorithms like Myers’ SES and RWS‑based techniques) to produce usable outputs (such as s‑expressions, JSON symbol lists, or protocol buffer representations).

##### Main Components and Their Roles
- Client Interface / CLI:
– Represents the command line tool entry point (see usage examples in the README).
– Handles flags (e.g. ––sexpression, ––json-symbols, ––proto-symbols) and dispatches commands.
- Input Layer:
– Responsible for reading source code either from the filesystem or from network/protocol buffer requests.
– Abstracts “blobs” as raw data.
- Parser Engine:
– Uses tree‑sitter grammar definitions to generate per‑language Haskell syntax types.
– Includes multiple language-specific modules (directories like semantic-go, semantic-java, semantic-python, etc.) that encapsulate parsing logic.
- Analysis Engine:
– Processes the generated Abstract Syntax Trees (ASTs) with sophisticated algorithms.
– Integrates diff algorithms (e.g. Myers’ O(ND) algorithm, RWS‑diff) and techniques such as open unions and data types à la carte.
– Often contains modules under directories such as semantic-analysis and semantic-tags.
- Output Renderer:
– Converts internal representations (blobs, ASTs, symbol lists) into one or more output formats.
– Supports s‑expressions, JSON, and protocol buffers (proto‑symbols).
- Build and Integration Infrastructure:
– The project shows support for both Cabal and Bazel build systems (with scripts like bootstrap and bootstrap‑bazel).
– Contains directories for build configuration, continuous integration workflows, and tooling.

##### Relationships and Interactions
- The client (CLI) passes user requests into the input layer.
- The input layer feeds raw code data into the parser engine.
- The parser engine, using tree‑sitter integrations, converts code into language‑specific ASTs.
- These ASTs are then handed over to the analysis engine where algorithms (e.g. diff, change detection, scope graph building) run over them.
- Finally, the output renderer formats the analysis results into the requested format for display or further processing.

##### Architectural Patterns and Design Principles
- Modular Design: The repository’s structure (with many “semantic-…” subdirectories) shows a modular approach – each language may have its own parser and analysis routines but builds on a common framework.
- Separation of Concerns: Distinct modules for input, parsing, analysis, and output allow independent development and testing.
- Extensibility: The usage of techniques like “data types à la carte” and open unions ensures new language support or output formats can be integrated with minimal rework.
- Use of well-established algorithms: Implementation of recognized diff algorithms (Myers’ and RWS) underlines a design leaning on proven methods.

##### Relevant Technologies
- Haskell as the primary programming language.
- Tree‑sitter for language grammar definitions.
- Cabal and Bazel as build systems.
- Protocol buffers for some forms of I/O (e.g. proto‑symbols).
- Standard CLI libraries and possibly techniques for pretty‑printing and output formatting.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Runtime Components
    subgraph "Client Interface / CLI"
        CLI1["Main (Entry Point)"]
        CLI2["CLI Module"]
    end

    subgraph "Input Layer (Blob Handling)"
        I1["Blob Reader (IO)"]
        I2["Blob Abstraction"]
    end

    subgraph "Parser Engine"
        P1["TreeSitter Integration"]
        P2["Semantic-Go"]
        P3["Semantic-Java"]
        P4["Semantic-Python"]
        P5["Semantic-Ruby"]
    end

    subgraph "Analysis Engine"
        ACore["Analysis Core"]
        AConcrete["Analysis: Concrete"]
        ADeadCode["Analysis: DeadCode"]
        ATypecheck["Analysis: Typecheck"]
    end

    subgraph "Output Renderer"
        O1["Output Format"]
        O2["SExpr Precise"]
    end

    subgraph "Build and Integration Infrastructure"
        B1["Cabal Project"]
        B2["Bazel BUILD"]
        B3["WORKSPACE"]
        B4["Bootstrap Script"]
        B5["Bootstrap Bazel Script"]
        B6["Tooling Scripts"]
    end

    %% Data Flow Connections
    CLI1 -->|"dispatches"| I1
    CLI2 -->|"handles flags"| I1

    I1 -->|"raw code"| P1
    I2 --> P1

    P1 -->|"generates ASTs"| ACore

    ACore -->|"diff (Myers'/RWS)"| O1
    ACore --> O2

    %% Internal Analysis Engine connections
    ACore --> AConcrete
    ACore --> ADeadCode
    ACore --> ATypecheck

    %% Build Infrastructure supports the project
    B1 ---|"builds"| CLI1
    B2 ---|"builds"| CLI1
    B3 ---|"builds"| CLI1

    %% Classes for styling
    class CLI1,CLI2,I1,I2,P1,P2,P3,P4,P5,ACore,AConcrete,ADeadCode,ATypecheck,O1,O2 runtime
    class B1,B2,B3,B4,B5,B6 build

    %% Click Events for Client Interface / CLI
    click CLI1 "https://github.com/github/semantic/blob/main/semantic/app/Main.hs"
    click CLI2 "https://github.com/github/semantic/blob/main/semantic/src/Semantic/CLI.hs"

    %% Click Events for Input Layer
    click I1 "https://github.com/github/semantic/blob/main/semantic/src/Data/Blob/IO.hs"
    click I2 "https://github.com/github/semantic/blob/main/semantic/src/Data/Blob.hs"

    %% Click Events for Parser Engine
    click P1 "https://github.com/github/semantic/blob/main/semantic/src/Parsing/TreeSitter.hs"
    click P2 "https://github.com/github/semantic/tree/main/semantic-go"
    click P3 "https://github.com/github/semantic/tree/main/semantic-java"
    click P4 "https://github.com/github/semantic/tree/main/semantic-python"
    click P5 "https://github.com/github/semantic/tree/main/semantic-ruby"

    %% Click Events for Analysis Engine
    click ACore "https://github.com/github/semantic/tree/main/semantic-analysis/src/Analysis/"
    click AConcrete "https://github.com/github/semantic/blob/main/semantic-analysis/src/Analysis/Analysis/Concrete.hs"
    click ADeadCode "https://github.com/github/semantic/blob/main/semantic-analysis/src/Analysis/Analysis/DeadCode.hs"
    click ATypecheck "https://github.com/github/semantic/blob/main/semantic-analysis/src/Analysis/Analysis/Typecheck.hs"

    %% Click Events for Output Renderer
    click O1 "https://github.com/github/semantic/blob/main/semantic/src/Serializing/Format.hs"
    click O2 "https://github.com/github/semantic/blob/main/semantic/src/Serializing/SExpression/Precise.hs"

    %% Click Events for Build and Integration Infrastructure
    click B1 "https://github.com/github/semantic/blob/main/cabal.project"
    click B2 "https://github.com/github/semantic/blob/main/BUILD.bazel"
    click B3 "https://github.com/github/semantic/tree/main/WORKSPACE"
    click B4 "https://github.com/github/semantic/tree/main/script/bootstrap"
    click B5 "https://github.com/github/semantic/tree/main/script/bootstrap-bazel"
    click B6 "https://github.com/github/semantic/tree/main/script"

    %% Class Definitions for Colors
    classDef runtime fill:#bbf,stroke:#333,stroke-width:2px;
    classDef build fill:#fbb,stroke:#333,stroke-width:2px;
```
