How fmartinez09/java-wal works
Project Type and Purpose
• This is a standalone Java library/tool implementing a WAL storage engine (not a web app).
• Purpose: provide durable, append-only logging with recovery, segmentation, checksums, and an iterator API.
File Structure
• src/main/java/LogFile.java – core class handling append, read, fsync, page buffering, fragmentation, and segment rotation.
• src/main/java/Record.java – encapsulates a payload + metadata (length, CRC).
• src/test/java/TestLog.java – unit tests for append/read/recovery behavior.
• pom.xml – Maven build, declares dependencies (e.g., for CRC or logging).
README for Architectural Insights
• Binary record format: length header, data, CRC32 footer.
• Page buffering & configurable padding, fsync policies.
• Record fragmentation logic (FULL/FIRST/MIDDLE/LAST).
• Recovery scan on startup that truncates incomplete/corrupt pages.
• Segment rotation scheme (wal-000001.log, wal-000002.log, …).
• Iterator API for replay.
Main Components to Illustrate
• Client API Layer
– Exposes append(Record) and readNext() / iterator operations.
• WAL Core / Manager
– Coordinates page buffering, fragmentation, CRC32 calculation, and segment rotation.
– Implements fsync policy (Always / Never / Every N records / Time-based).
• Page Buffer & Serializer
– Serializes Record → [len][data][crc].
– Splits/fragments large records across fixed-size pages.
• File I/O & Segmentation
– Writes buffers to disk files named wal-000001.log, etc.
– Reads from appropriate segment during replay.
• Recovery Module
– On open, scans all segments, validates checksums, truncates incomplete pages/records.
• Underlying File System (OS)
– Durable storage; fsync calls for persistence guarantees.
Relationships & Data Flow
1. Client calls append() → WAL Manager
2. WAL Manager serializes via Page Buffer → applies fragmentation
3. Buffer flushed to File I/O layer → OS cache → optionally fsync
4. On readNext() or iterator, File I/O → Page Buffer → deserializer → Record to client
5. At startup/close, Recovery Module inspects segments → validates CRC → truncates if needed
Architectural Patterns & Principles
• Layered Architecture:
– API Layer → Core/Manager → I/O Layer → OS
• Single Responsibility:
– LogFile.java handles orchestration; Record.java handles data encapsulation.
• Pluggable Policies:
– FsyncPolicy interface (Always, Never, EveryN, TimeBased).
• Modular Recovery:
– Segregate startup scan & truncation logic.
Technologies / Frameworks
• Java SE (java.io.File, java.nio.*, java.util.zip.CRC32)
• Maven (pom.xml) for build and dependency management
• JUnit/TestNG (tests) for unit testing
[ ] Main components (API, Core, Page Buffer, File I/O, Recovery, Disk)
[ ] Key format details ([len|data|crc], fragmentation stages)
[ ] Configuration/policy hooks (fsync, page size)
[ ] Segment sequence representation
[ ] Recovery path on startup