How simd-lite/simd-json works
This project is a high-performance Rust library for JSON parsing that leverages hardware SIMD instructions. In essence, it is a port of the simdjson C++ library, redesigned to work naturally in the Rust ecosystem. The core purpose is to parse JSON data as quickly as possible by using different optimized code paths (AVX2, SSE4.2, NEON, SIMD128) while also offering familiar APIs for both Serde and a DOM-like interface.
And delineate the main components
• Core Parsing Engine – Show the central “Parser” that receives raw JSON bytes and produces a parsed representation.
• SIMD Optimized Implementations – Lay out the various modules under src/impls:
• AVX2
• SSE4.2
• NEON
• SIMD128
• Native / fallback implementation
• API Layers – Identify the different usage entry points:
• Values API (Borrowed and Owned Values, implemented under src/value and related directories)
• Tape API (responsible for converting JSON to an intermediate “tape” representation in src/value/tape)
• Serde Compatible API (located in src/serde, enabling integration with Serde)
• Auxiliary Components – Include error handling (modules like src/error.rs), number parsing (in src/numberparse), string parsing (src/stringparse, charutils) and portability support.
Relationships and interactions
• Within the Core Parsing Engine, use a decision diamond to direct execution to the appropriate SIMD module (AVX2, SSE42, NEON, SIMD128) based on platform detection (enabled by the “runtime-detection” feature) or to a fallback native implementation.
• For experimental or auxiliary modules (like fuzzing targets in the fuzz folder and benchmark tests in benches), include a side note or peripheral components block indicating testing/validation subsystems.
Architectural patterns and important design principles
• Component separation: Clearly show a layered design where the API layer is separated from the core SIMD-optimized parsing logic.
• Runtime detection: Represent with a decision or switch mechanism that selects the appropriate optimized backend based on the detected CPU features.
• Safety vs. performance trade-offs: Note where “unsafe” code is used (primarily in SIMD intrinsics) and where additional unit tests and fuzzing in tests/ and fuzz/ ensure resilience.
• Feature toggles: Indicate optional features such as “serde_impl”, “swar-number-parsing”, “known-key”, etc., that modify behavior or integrate additional libraries.
Relevant details on technologies and frameworks
• Rust and the Cargo build system including the project’s Cargo.toml.
• Integration with build and CI/CD: The .github and .drone files indicate automated testing and quality checks.
• Dependency on specific allocators (mimalloc, jemalloc, or snmalloc) for better performance.
• Use of SIMD intrinsics: Mark the modules (in src/impls) that wrap hardware-specific instructions.
• Clear labels for each major component (for example: “API Entry Points”, “Runtime SIMD Detection”, “AVX2 Implementation”, “Serde Adapter”).