# sourcegraph/scip architecture

> Architecture diagram of the GitHub repository sourcegraph/scip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sourcegraph/scip
- Repository: https://github.com/sourcegraph/scip

## Overview

##### Project Type and Purpose
- This is an open‑source, language‑agnostic code‑intelligence protocol implementation and toolkit.
- Core purpose: define a Protobuf schema (`scip.proto`) for source‑code indexing (SCIP), provide a CLI (`cmd/scip`) to work with SCIP indexes, and offer rich language bindings (Go, Rust) plus generated bindings (TypeScript, Haskell).
- Use cases: generating SCIP indexes in various indexers, consuming SCIP data in tooling or UIs, and using the `scip` CLI for inspection, linting, printing, snapshot testing, etc.

##### File Structure Analysis
- Top level:
- `scip.proto` + Buf files (`buf.yaml`, `buf.gen.yaml`): core schema and codegen config.
- `bindings/`: language‑specific modules—
– `go`: manual bindings + helpers/tests (internal parsing, symbol tables, snapshots)
– `rust`: Cargo project with generated `.rs` + symbol helpers
– `typescript`: auto‑gen `.js`/`.ts` + `package.json`
– `haskell`: Cabal project, auto‑gen modules
- `cmd/scip/`: CLI implementation in Go, with subcommands (lint, print, stats, snapshot, test) + test suites (reprolang, snapshots, test_cmd).
- `dev/`: Dockerfiles and scripts for building/testing in containers; codegen scripts (`proto-generate.sh`); release scripts.
- CI/CD: `.github/workflows/` for multi‑language builds (Go, Rust, Haskell, TypeScript), formatting, release automation.
- `docs/`: CLI reference, protocol design, file format spec.

##### Key Insights from README & DESIGN.md
- The protocol is designed to be embedded in many language indexers; language bindings wrap the Protobuf objects and add utilities (flattening, sanitization, symbol formatting).
- The CLI is a standalone Go binary that consumes/generated SCIP files.
- Build/test pipelines ensure codegen is up to date and language modules compile.

a. Main Components
1) Core Schema & Codegen Layer
- `scip.proto` + Buf toolchain → generates language outputs
2) Language Binding Libraries
- Go binding module
- Rust binding crate
- TypeScript package
- Haskell package
3) `scip` CLI
- Subcommands: lint, print, stats, test, snapshot
- Relies on Go binding
4) External Indexers (e.g., scip-java, rust-analyzer)
- Consume codebases → emit `.scip` protobuf indexes
5) CI/CD & Dev Infrastructure
- Docker environments, GitHub Actions pipelines, release scripts
6) Consumers of SCIP Data
- UIs (Sourcegraph), other tooling via bindings

b. Component Interactions
- Codegen: Buf reads `scip.proto` → emits language artifacts into each `bindings/` folder.
- Language bindings: provide APIs to parse, manipulate, and write SCIP indexes.
- External indexers link against bindings or use Protobuf tooling to emit `.scip` files.
- `scip` CLI uses Go binding to read `.scip` files, apply commands, and write output.
- CI pipelines run codegen, compile/test each binding module, build CLI, run snapshot tests.
- Dev scripts (Docker) orchestrate containerized builds/testing for reproducibility.

c. Architectural Patterns & Principles
- Modular monorepo: one schema, multiple language modules.
- Code generation as first‑class build step.
- Clear separation between schema (protocol), language SDKs (bindings), and tooling (CLI).
- Test harnesses organized by module (`bindings/go`, `cmd/scip/tests`).
- Infrastructure as code: CI workflows and Docker scripts ensure consistency.

d. Key Technologies
- Protobuf + Buf for schema management & codegen
- Go (CLI, primary binding), Rust (binding), TypeScript, Haskell
- Docker for controlled build envs
- GitHub Actions for CI/CD
- Shell scripts for release, codegen, publishing

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Schema & Codegen Layer
    subgraph "Schema & Codegen Layer"
        direction TB
        SchemaProto["scip.proto"]:::schema
        BufConfig["buf.yaml / buf.gen.yaml / buf.work.yaml"]:::schema
    end

    %% Language Binding Libraries
    subgraph "Language SDKs"
        direction TB
        GoBindings["Go Binding (bindings/go/scip/)"]:::binding
        RustBindings["Rust Binding (bindings/rust/)"]:::binding
        TSBindings["TypeScript Binding (bindings/typescript/)"]:::binding
        HsBindings["Haskell Binding (bindings/haskell/)"]:::binding
    end

    %% CLI Tool
    SCIPCLI["scip CLI (cmd/scip/)"]:::cli

    %% External Ecosystem
    subgraph "External Ecosystem"
        direction TB
        ExtIndexers["External Indexers"]:::external
        Consumers["Consumers (UIs, Tools)"]:::external
    end

    %% Infrastructure
    subgraph "CI/CD & Dev Infrastructure"
        direction TB
        CI["GitHub Actions & Workflows"]:::infra
        DevScripts["Dev Scripts & Docker"]:::infra
    end

    %% Documentation & Version
    Docs["Documentation (docs/)"]:::docs
    Version["version.txt"]:::docs

    %% Connections
    SchemaProto -->|"defines schema for"| GoBindings
    SchemaProto -->|"defines schema for"| RustBindings
    SchemaProto -->|"defines schema for"| TSBindings
    SchemaProto -->|"defines schema for"| HsBindings
    BufConfig -->|"generates code into"| GoBindings
    BufConfig -->|"generates code into"| RustBindings
    BufConfig -->|"generates code into"| TSBindings
    BufConfig -->|"generates code into"| HsBindings

    GoBindings -->|"used by"| SCIPCLI
    ExtIndexers -->|"emit .scip protobuf"| SCIPCLI
    ExtIndexers -->|"emit .scip protobuf"| GoBindings
    SCIPCLI -->|"inspect / lint / print"| Consumers
    GoBindings -->|"provide APIs to"| Consumers

    CI -->|"runs codegen & tests for"| GoBindings
    CI -->|"runs codegen & tests for"| RustBindings
    CI -->|"runs codegen & tests for"| TSBindings
    CI -->|"runs codegen & tests for"| HsBindings
    CI -->|"builds & packages"| SCIPCLI
    DevScripts -->|"orchestrate builds| tests"| CI
    DevScripts -->|"support local dev"| GoBindings

    Docs -->|"reference for"| SCIPCLI
    Docs -->|"spec for"| SchemaProto
    Version -->|"package version for"| SCIPCLI

    %% Click Events
    click SchemaProto "https://github.com/sourcegraph/scip/blob/main/scip.proto"
    click BufConfig "https://github.com/sourcegraph/scip/blob/main/buf.yaml"
    click GoBindings "https://github.com/sourcegraph/scip/tree/main/bindings/go/scip/"
    click RustBindings "https://github.com/sourcegraph/scip/tree/main/bindings/rust/"
    click TSBindings "https://github.com/sourcegraph/scip/tree/main/bindings/typescript/"
    click HsBindings "https://github.com/sourcegraph/scip/tree/main/bindings/haskell/"
    click SCIPCLI "https://github.com/sourcegraph/scip/tree/main/cmd/scip/"
    click CI "https://github.com/sourcegraph/scip/tree/main/.github/workflows/"
    click DevScripts "https://github.com/sourcegraph/scip/tree/main/dev/"
    click Docs "https://github.com/sourcegraph/scip/tree/main/docs/"
    click Version "https://github.com/sourcegraph/scip/blob/main/version.txt"

    %% Styles
    classDef schema fill:#d0e1f9,stroke:#0366d6,color:#0366d6,stroke-width:1px;
    classDef binding fill:#daf5d8,stroke:#22863a,color:#22863a,stroke-width:1px;
    classDef cli fill:#ffe8d6,stroke:#e36209,color:#e36209,stroke-width:1px;
    classDef external fill:#f1f1f1,stroke:#6a737d,color:#6a737d,stroke-width:1px;
    classDef infra fill:#f0dbf4,stroke:#6f42c1,color:#6f42c1,stroke-width:1px;
    classDef docs fill:#fafbfc,stroke:#24292e,color:#24292e,stroke-width:1px;
```
