How apache/fory works
Project type and purpose
• This is a multi-language serialization framework (“Apache Fory™”) that provides high-performance, cross-language object serialization/deserialization and a zero-copy row format.
• It ships implementations in Java, C++, Python, Go, Rust, JavaScript, Kotlin, Scala, and Dart—each following a common protocol specification (Xlang, Java-native, Row Format, etc.).
File structure
• Top-level directories correspond to language implementations: java/, cpp/, python/, go/, rust/, javascript/, kotlin/, scala/, dart/.
• A bazel/ directory and root BUILD/WORKSPACE indicate a Bazel-based, polyglot build setup (for C++, Python, Java, etc.).
• CI & release scripts under .github/ and ci/ coordinate multi-language builds, tests, packaging, and deployments.
• Shared documentation in docs/ and integration tests in integration_tests/ demonstrate cross-language compatibility.
README for additional insights
• The README emphasizes core features: JIT compilation (Java), static codegen (C++, Rust, Go), zero-copy row format, meta-sharing, and cross-language protocols.
• It lists protocol specifications (Xlang, Java Serialization, Row Format) and user guides per language, implying that each language implementation must adhere to the central specifications.
Main components and how they interact
a. Monorepo Root
• Contains global build definitions (Bazel, Maven/Gradle POMs, Cargo.toml, setup.py, package.json).
• Houses CI workflows that orchestrate building, testing, and packaging each language artifact.
• Java: Modules fory-core, fory-format, fory-simd, fory-extensions, test suites. Uses Maven/Gradle, JIT compilation (Janino), GraalVM native-image.
• Python: Under python/pyfory: Cython wrappers around the C++ core plus pure-Python fallbacks. Implements the same protocols via _serializer, _struct, format, etc.
• Go: Under go/fory: idiomatic Go implementation with static code-generation via go generate.
• Rust: Under rust/: fory-core, fory-derive (proc-macros), benches/tests.
• JavaScript: Under javascript/packages/fory: TypeScript implementation plus an optional native extension (hps).
• Kotlin, Scala, Dart: Lightweight wrappers on the Java or native core, providing language-specific serializers.
c. Protocol Specifications & Codegen
• The Xlang Serialization Spec and Row Format Spec (under docs/specification/) define the wire formats and type-mapping rules.
• Each implementation uses either code generation (Java JIT, Rust derive macro, Go generator, C++ templates) or dynamic resolution (Python, JS) to produce efficient serialization/deserialization code conforming to these specs.
d. Integration & Testing
• integration_tests/ contains cross-language test suites and benchmarks (e.g., cpython_benchmark, graalvm_tests, JDK compatibility tests).
• Each language’s test/ folder verifies compliance with the specs and measures performance.
Architectural patterns and design principles
• Polyglot Modularization: Each language lives in its own directory/module, but they share a unified specification.
• Code Generation vs. Dynamic: Static codegen (Rust Proc-Macros, Go generator, C++ templates) vs. dynamic resolution (Python Cython, JS TypeScript).
• Separation of Concerns: Build logic separated from language-specific logic; protocol spec separated from implementation.
• Micro-modules: Java approach uses small Maven modules (fory-core, fory-format, etc.) to allow optional features (SIMD, format).