How buger/probe works
Project Type & Purpose
• Project type: Full‐stack, polyglot tool comprising
– A Rust core (CLI binary + library) for high‐performance code search and extraction
– Multiple JavaScript/TypeScript “wrappers” (npm CLI, MCP server agent, chat & web UI examples)
– Documentation website (VitePress)
• Purpose: Local, semantic code search powered by ripgrep (fast scanning) + tree‐sitter (AST parsing) + NLP/ranking (BM25, TF-IDF, hybrid). It exposes CLI commands, an MCP server, and JS/Node SDKs for AI integrations and web interfaces.
File Structure Analysis
Top‐level:
– Cargo.toml & src/… : Rust core modules
– examples/ : JS examples (chat, token counters, web server)
– mcp-agent/ & mcp/ & npm/ : Node.js packages
– site/ : docs site (VitePress, Vue components)
– Makefile, GitHub Actions workflows: build/test/release pipelines
Rust core (src/):
– cli.rs, main.rs, lib.rs – entrypoints
– search/, extract/, query.rs, ranking.rs – pipeline stages
– language/ – per‐language parser implementations (implements LanguageTrait)
– path_resolver/ – language‐specific import/path logic
– models.rs – shared data structures
Node wrappers:
– mcp-agent/: “@buger/probe-mcp” agent for MCP protocol
– mcp/: higher‐level JS binding for MCP server
– npm/: “@buger/probe” npm CLI package (downloads binary or bundles it)
Docs site:
– site/.vitepress – config & custom components
– Markdown under site/ – guides, use cases
README Insights
• Modes of operation: CLI, MCP Server, AI Chat Mode, Web Interface
• External libraries: ripgrep, tree‐sitter, Node.js/ai SDKs
• CI/CD: GitHub Actions for Rust tests, release, site deployment
Main Components & Interactions
a. Core Rust Engine
– Input: CLI args or JSON via MCP or child_process stdio
– Pipeline stages:
1) Ripgrep Scanner (shell out to ripgrep)
2) AST Parser (tree‐sitter integration)
3) Tokenizer & Search (token_utils, search/)
4) Re‐ranking (ranking.rs: BM25, TF-IDF, hybrid)
5) Extractor (extract/; formatter)
6) Output Formatter (JSON, Markdown, plain)
– Modules communicate via in‐process function calls and shared data models
b. Node.js Wrappers & Frontends
– npm CLI (“probe”): calls binary or downloads via postinstall
– MCP Agent (“@buger/probe-mcp”): spawns binary in server mode, exposes MCP protocol
– AI Chat Example (examples/chat & npm package “@buger/probe-chat”): uses ProbeChat SDK → invokes binary via child_process or HTTP to MCP → streams responses
– Web Interface (“@buger/probe-web” example): browser UI + Node.js server (probeTool.js) → invokes CLI/MCP, renders with Markdown/Highlight.js
c. Documentation Site
– VitePress‐based site, served statically on GitHub Pages
– No runtime interplay with core binary
Architectural Patterns & Design Principles
• Modular pipeline: clear separation of scanning, parsing, searching, ranking, extracting
• Strategy/Factory Pattern: LanguageTrait implementations in src/language, registered via factory.rs
• IPC/Process‐boundary interfaces: Rust binary exposes CLI, JSON, and MCP‐compatible protocols
• Wrapper layering: thin JS/TS layers over core binary to enable multiple deployment scenarios (CLI, server, SDK)
• CI/CD: GitHub Actions workflows build/test/deploy each component independently