How rsonquery/rsonpath works
Project type and purpose
• This is a Rust-based, SIMD-accelerated JSONPath engine and CLI tool (“rq”) composed of multiple interdependent crates:
• rsonpath – the command-line executable and runner
• rsonpath-lib – the core query compilation and execution engine
• rsonpath-syntax – the query parser
• rsonpath-benchmarks, rsonpath-test, rsonpath-test-codegen – performance and compliance test harnesses
• book – documentation site generated by mdBook
• Purpose: parse JSONPath queries, compile them to a fast automaton, execute against streamed or memory-mapped JSON, output matched values or counts, all with SIMD acceleration where available.
File structure
• Top-level directories:
• crates/… – library and tool crates
• book/… – user/developer book source
• fuzz/… – fuzz targets for ClusterFuzzLite
• .github/… – CI, fuzz, release workflows
• Key config files: Cargo.toml (workspace), Justfile (build/test script), .clusterfuzzlite (fuzz CI), rust-toolchain.toml, shell.nix.
• Sub-crates indicate a layered architecture: syntax → lib → CLI.
Extract additional insights from README
• Notes for SIMD feature flags, supported CPU targets.
• Benchmarks and documentation generation processes.
a. Main components
• User & Environment:
• Terminal / Std-in / File system
• CLI (rsonpath crate using clap, simple-logger)
• Query pipeline:
• Parser (rsonpath-syntax crate, nom)
• Compiler (automaton builder in rsonpath-lib)
• Execution Engine (rsonpath-lib: engine modules for head/tail skipping, classification, NFA, SIMD kernels)
• Data I/O:
• Input reader (memmap2 or buffered reader module)
• Output serializer (JSON serializer or line-delimited)
• Support tooling:
• Benchmarks (rsonpath-benchmarks)
• Compliance tests (rsonpath-test, rsonpath-syntax-proptest)
• Docs generator (mdBook in book/)
• CI / Fuzzing:
• GitHub workflows
• ClusterFuzzLite (.clusterfuzzlite)
b. Relationships & data flow
1. User invokes rq CLI
2. CLI parses flags & query string (clap)
3. CLI calls Parser → produces AST
4. Compiler transforms AST → automaton (vector-map, smallvec)
5. Engine executes automaton over JSON input (memmap2, SIMD modules)
6. Engine emits matches → Serializer → Stdout or file
7. Optional paths: benchmarking harness invokes engine directly; fuzz targets feed arbitrary input into parser & engine.
c. Architectural patterns & principles
• Layered modular design: parser → compiler → executor
• Feature flags for conditional SIMD modules (cfg-if)
• Separation of concerns: CLI vs. core library
• Reusable crates for testing and benchmarking
d. Key technologies & libraries
• Rust crates: clap, nom, memmap2, smallvec, vector-map, thiserror, eyre
• SIMD acceleration: AVX2/SSE modules under classification and engine in rsonpath-lib
• Build/test: cargo + Just, GitHub Actions, ClusterFuzzLite, shell.nix