# buger/probe architecture

> Architecture diagram of the GitHub repository buger/probe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/buger/probe
- Repository: https://github.com/buger/probe

## Overview

##### Project Type & Purpose
- Project type: Full‐stack, polyglot tool comprising
– A Rust core (CLI binary + library) for high‐performance code search and extraction
– Multiple JavaScript/TypeScript “wrappers” (npm CLI, MCP server agent, chat & web UI examples)
– Documentation website (VitePress)
- Purpose: Local, semantic code search powered by ripgrep (fast scanning) + tree‐sitter (AST parsing) + NLP/ranking (BM25, TF-IDF, hybrid). It exposes CLI commands, an MCP server, and JS/Node SDKs for AI integrations and web interfaces.

##### File Structure Analysis
Top‐level:
– Cargo.toml & src/… : Rust core modules
– examples/ : JS examples (chat, token counters, web server)
– mcp-agent/ & mcp/ & npm/ : Node.js packages
– site/ : docs site (VitePress, Vue components)
– Makefile, GitHub Actions workflows: build/test/release pipelines
Rust core (src/):
– cli.rs, main.rs, lib.rs – entrypoints
– search/, extract/, query.rs, ranking.rs – pipeline stages
– language/ – per‐language parser implementations (implements LanguageTrait)
– path_resolver/ – language‐specific import/path logic
– models.rs – shared data structures
Node wrappers:
– mcp-agent/: “@buger/probe-mcp” agent for MCP protocol
– mcp/: higher‐level JS binding for MCP server
– npm/: “@buger/probe” npm CLI package (downloads binary or bundles it)
Docs site:
– site/.vitepress – config & custom components
– Markdown under site/ – guides, use cases

##### README Insights
- Modes of operation: CLI, MCP Server, AI Chat Mode, Web Interface
- External libraries: ripgrep, tree‐sitter, Node.js/ai SDKs
- CI/CD: GitHub Actions for Rust tests, release, site deployment

##### Main Components & Interactions
a. Core Rust Engine
– Input: CLI args or JSON via MCP or child_process stdio
– Pipeline stages:
1) Ripgrep Scanner (shell out to ripgrep)
2) AST Parser (tree‐sitter integration)
3) Tokenizer & Search (token_utils, search/)
4) Re‐ranking (ranking.rs: BM25, TF-IDF, hybrid)
5) Extractor (extract/; formatter)
6) Output Formatter (JSON, Markdown, plain)
– Modules communicate via in‐process function calls and shared data models
b. Node.js Wrappers & Frontends
– npm CLI (“probe”): calls binary or downloads via postinstall
– MCP Agent (“@buger/probe-mcp”): spawns binary in server mode, exposes MCP protocol
– AI Chat Example (examples/chat & npm package “@buger/probe-chat”): uses ProbeChat SDK → invokes binary via child_process or HTTP to MCP → streams responses
– Web Interface (“@buger/probe-web” example): browser UI + Node.js server (probeTool.js) → invokes CLI/MCP, renders with Markdown/Highlight.js
c. Documentation Site
– VitePress‐based site, served statically on GitHub Pages
– No runtime interplay with core binary

##### Architectural Patterns & Design Principles
- Modular pipeline: clear separation of scanning, parsing, searching, ranking, extracting
- Strategy/Factory Pattern: LanguageTrait implementations in src/language, registered via factory.rs
- IPC/Process‐boundary interfaces: Rust binary exposes CLI, JSON, and MCP‐compatible protocols
- Wrapper layering: thin JS/TS layers over core binary to enable multiple deployment scenarios (CLI, server, SDK)
- CI/CD: GitHub Actions workflows build/test/deploy each component independently

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1: Rust Core Engine
    subgraph "Rust Core Engine"
        direction TB
        CLI["Core Rust CLI Binary"]:::rust
        LIB["Core Rust Library Entrypoint"]:::rust
        CLI_ARG["CLI Argument Parsing & Commands"]:::rust
        MODELS["Shared Data Models"]:::rust
        subgraph "Language & Path"
            direction TB
            FACTORY["Language Strategy/Factory"]:::rust
            TRAIT["Language Trait Definitions"]:::rust
            PARSERS["Language Implementations"]:::rust
            PATH_RESOLVER["Path Resolver Logic"]:::rust
        end
        subgraph "Pipeline"
            direction TB
            SCANNER["Ripgrep Scanner"]:::external
            AST["AST Parser"]:::external
            TOKEN_SEARCH["Tokenizer & Search Utilities"]:::rust
            RERANK["Re-ranking Module"]:::rust
            EXTRACT["Extraction Pipeline"]:::rust
            OUTPUT_FMT["Output Formatting"]:::rust
        end

        CLI -->|"calls"| LIB
        CLI --> CLI_ARG
        CLI_ARG --> SCANNER
        SCANNER --> AST
        AST --> TOKEN_SEARCH
        TOKEN_SEARCH --> RERANK
        RERANK --> EXTRACT
        EXTRACT --> OUTPUT_FMT
        LIB --> MODELS
        LIB --> FACTORY
        FACTORY --> TRAIT
        FACTORY --> PARSERS
        LIB --> PATH_RESOLVER
        AST --> FACTORY
        TOKEN_SEARCH --> MODELS
        EXTRACT --> MODELS
    end

    %% Layer 2: Node.js Wrappers & Frontends
    subgraph "Node.js Wrappers & Frontends"
        direction TB
        NPM_CLI["npm CLI Wrapper"]:::wrapper
        MCP_AGENT["MCP Agent Wrapper"]:::wrapper
        MCP_CLIENT["MCP Client Library"]:::wrapper
        CHAT_SDK["AI Chat SDK Example"]:::wrapper
        WEB_UI["Web Interface Example"]:::wrapper

        NPM_CLI -->|"spawns binary"| CLI
        MCP_AGENT -->|"MCP protocol"| CLI
        MCP_CLIENT -->|"MCP over HTTP"| MCP_AGENT
        CHAT_SDK -->|"child_process stdio"| CLI
        WEB_UI -->|"HTTP API"| MCP_AGENT
    end

    %% Layer 3: Documentation & CI/CD
    subgraph "Documentation & CI/CD"
        direction TB
        DOCS["Documentation Website"]:::docsci
        CI["CI/CD Workflows"]:::docsci

        CI --> DOCS
    end

    %% Click Events
    click CLI "https://github.com/buger/probe/blob/main/src/main.rs"
    click LIB "https://github.com/buger/probe/blob/main/src/lib.rs"
    click CLI_ARG "https://github.com/buger/probe/blob/main/src/cli.rs"
    click SCANNER "https://github.com/buger/probe/blob/main/src/search/file_processing.rs"
    click AST "https://github.com/buger/probe/blob/main/src/language/parser.rs"
    click FACTORY "https://github.com/buger/probe/blob/main/src/language/factory.rs"
    click TRAIT "https://github.com/buger/probe/blob/main/src/language/language_trait.rs"
    click PATH_RESOLVER "https://github.com/buger/probe/blob/main/src/path_resolver/mod.rs"
    click TOKEN_SEARCH "https://github.com/buger/probe/blob/main/src/search/token_utils.rs"
    click TOKEN_SEARCH "https://github.com/buger/probe/blob/main/src/search/search_runner.rs"
    click RERANK "https://github.com/buger/probe/blob/main/src/ranking.rs"
    click EXTRACT "https://github.com/buger/probe/blob/main/src/extract/mod.rs"
    click OUTPUT_FMT "https://github.com/buger/probe/blob/main/src/search/results_formatter.rs"
    click OUTPUT_FMT "https://github.com/buger/probe/blob/main/src/extract/formatter.rs"
    click MODELS "https://github.com/buger/probe/blob/main/src/models.rs"
    click NPM_CLI "https://github.com/buger/probe/blob/main/npm/src/cli.js"
    click NPM_CLI "https://github.com/buger/probe/blob/main/npm/scripts/postinstall.js"
    click MCP_AGENT "https://github.com/buger/probe/blob/main/mcp-agent/src/agent.js"
    click MCP_AGENT "https://github.com/buger/probe/blob/main/mcp-agent/src/index.js"
    click MCP_CLIENT "https://github.com/buger/probe/blob/main/mcp/src/index.ts"
    click CHAT_SDK "https://github.com/buger/probe/blob/main/examples/chat/bin/probe-chat.js"
    click CHAT_SDK "https://github.com/buger/probe/blob/main/examples/chat/probeChat.js"
    click WEB_UI "https://github.com/buger/probe/blob/main/examples/chat/webServer.js"
    click WEB_UI "https://github.com/buger/probe/blob/main/examples/chat/probeTool.js"
    click DOCS "https://github.com/buger/probe/blob/main/site/.vitepress/config.mts"
    click DOCS "https://github.com/buger/probe/blob/main/site/quick-start.md"
    click DOCS "https://github.com/buger/probe/blob/main/site/mcp-server.md"
    click CI "https://github.com/buger/probe/blob/main/.github/workflows/rust-tests.yml"
    click CI "https://github.com/buger/probe/blob/main/.github/workflows/release.yml"
    click CI "https://github.com/buger/probe/blob/main/.github/workflows/vitepress-gh-pages.yml"

    %% Styles
    classDef rust fill:#a8d5a2,stroke:#333,stroke-width:1px;
    classDef wrapper fill:#9ec5e8,stroke:#333,stroke-width:1px;
    classDef external fill:#f4a261,stroke:#333,stroke-width:1px;
    classDef docsci fill:#bbbbbb,stroke:#333,stroke-width:1px;
```
