How facebook/pyrefly works
Project Type and Purpose
• Pyrefly is a multi‐platform, full‐stack static type checker and IDE for Python.
• It consists of a core type‐checking engine in Rust (library and CLI), a Language Server Protocol (LSP) extension for editors (VSCode), a WebAssembly build for in‐browser sandboxing, a Python‐package wrapper (pip), a Docusaurus‐based documentation website, and a conformance‐test suite.
File‐Structure Analysis
• Top‐level Rust crates under pyrefly/, pyrefly_wasm/, pyrefly_derive/, pyrefly_util/.
• LSP extension in lsp/ (TypeScript + Node.js).
• Documentation site in website/ (Docusaurus, React).
• Conformance tests in conformance/ and test/.
• CI/CD workflows under .github/workflows/ for building binaries, publishing to PyPI, deploying website and extension.
Key Components
a. Core Engine (Rust crates)
• pyrefly (binding, graph, solver, state, module, config, commands)
• pyrefly_util (utility crate)
• pyrefly_derive (derive macros)
b. CLI & Python Wrapper
• pyrefly/bin/main.rs (CLI front-end)
• pyrefly.dotslash.py & pyproject.toml (Python pip package)
c. Language Server & Editor Extension
• LSP server hook in pyrefly (command: lsp.rs)
• VSCode extension in lsp/src/extension.ts
d. Web Frontend & WASM Sandbox
• pyrefly_wasm crate → Compiles engine to WebAssembly
• website/src/sandbox/* → React components + pythonWorker.ts
• Docusaurus site in website/ (docs, sandbox launcher)
e. Conformance Test Suite
• Python scripts in conformance/ driving the engine
• scripts/fetch_typeshed.py and test/glean/ JSON fixtures
f. CI/CD Pipelines
• GitHub Actions flows for building/testing/deploying all artifacts
Relationships and Data Flows
• The Core Engine (Rust crates) exposes an API used by three “clients”:
1. CLI & Python Wrapper (invoked from shell or pip install pyrefly)
2. LSP Server (spawned by VSCode extension)
3. WASM Module (embedded in the website sandbox)
• The VSCode Extension communicates over the LSP protocol to a server process that delegates to the engine.
• The Website loads the WASM module into the browser sandbox; user code is type‐checked in-browser.
• Conformance tests feed Python snippets to the CLI or engine API and verify JSON/console outputs.
• CI/CD orchestrates builds: Rust binaries, Python wheels, WASM artifacts, npm packages, and documentation deployment.
Architectural Patterns and Design Principles
• Modular “crate” structure in Rust for separation of concerns (binding, solving, state management).
• Command pattern for CLI commands (init, check, run, lsp).
• Single‐module incrementality: each Python module is fully bound and solved in isolation, then combined.
• Polyglot clients: the same engine library supports native CLI, LSP, WebAssembly, and Python binding.
• Conventional separation of front‐end (website + VSCode UI) and back‐end (engine).
Technologies, Frameworks, and Libraries
• Rust & Cargo (core engine, CLI, WASM)
• wasm-pack / wasm-bindgen (WASM build)
• Python packaging (pip, pyproject.toml)
• Node.js, TypeScript, ESLint (VSCode extension)
• Docusaurus, React (documentation site)
• GitHub Actions (CI/CD)
• Third-party typeshed stubs under third_party/typeshed for standard library typing
Detailed Capture of Essential Elements
• Multi‐client architecture (CLI, LSP, WASM, Python).
• Website both as documentation and as host for the sandbox.
• CI/CD flows driving artifact production and publishing (PyPI, npm, website host).
• Conformance test subsystem invoking the engine via Python scripts.
• Every component from the file tree (crates, directories) is placed in context to reflect actual dependencies.