# nubskr/walrus architecture

> Architecture diagram of the GitHub repository nubskr/walrus, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nubskr/walrus
- Repository: https://github.com/nubskr/walrus

## Overview

##### Project type and purpose
- Walrus is a Rust library/crate that provides a high-performance, topic-based write-ahead log (WAL) storage engine. It’s not a full-stack or web service but a standalone storage component intended for embedding in applications or benchmarking against systems like Kafka and RocksDB.

##### File structure
- Top-level:
- Cargo.toml / src/lib.rs – public API entrypoint
- src/wal/… – core engine modules (block.rs, config.rs, paths.rs, storage.rs)
- src/wal/runtime/… – runtime submodules (allocator, background tasks, index, reader, writer, walrus orchestration)
- docs/ – narrative architecture & design deep-dives
- benchmarks/ & scripts/ – performance test harness and visualization tools
- tests/ – unit, integration, end-to-end, rollback and concurrency tests
- Build & CI: Makefile plus GitHub workflows for CI, benchmarks, scaling tests.

##### README for additional insights
- Key features: configurable consistency (strict vs at-least-once), fsync scheduling, dual backends (pread/pwrite + io_uring vs mmap), atomic batch I/O, topic isolation, persistent offsets, coordination-free cleanup.
- The project already contains an architecture doc under docs/architecture.md.

##### Diagramming conventions and elements
- Clear labels for every module/component (e.g., “Allocator”, “Writer”, “Storage Backend”)
- Blue rectangles for public/API components
- Green rectangles for runtime modules
- Notes or callouts for environment-driven configuration and consistency options

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    Client["Client Application"]:::external

    %% Public/API Layer
    subgraph "Public/API Layer"
        API["Walrus (src/lib.rs)"]:::api
        ModExports["Public API exports (src/wal/mod.rs)"]:::api
    end

    %% Configuration & Topic Paths
    subgraph "Configuration & Topic Management"
        Config["Configuration Manager (src/wal/config.rs)"]:::api
        Paths["Topic Manager & Paths (src/wal/paths.rs)"]:::api
    end

    %% Runtime Subsystems
    subgraph "Runtime Subsystems"
        Allocator["Allocator (src/wal/runtime/allocator.rs)"]:::runtime
        WriterCore["Writer core (src/wal/runtime/writer.rs)"]:::runtime
        BatchWrite["Batch write glue (src/wal/runtime/walrus_write.rs)"]:::runtime
        ReaderCore["Reader core (src/wal/runtime/reader.rs)"]:::runtime
        BatchRead["Batch read glue (src/wal/runtime/walrus_read.rs)"]:::runtime
        BatchOrch["Batch Orchestrator (src/wal/runtime/walrus.rs)"]:::runtime
        Background["Background Tasks & Checkpointing (src/wal/runtime/background.rs)"]:::runtime
        IndexMgr["Index Manager (src/wal/runtime/index.rs)"]:::runtime
    end

    %% Storage Engine Core
    subgraph "Storage Engine Core"
        BlockMgr["Block Manager (src/wal/block.rs)"]:::api
        StorageBackend["Storage Backend Abstraction (src/wal/storage.rs)"]:::storage
        WalFile["Persistent WAL File"]:::storage
    end

    %% OS Subsystems
    subgraph "OS Subsystems"
        IOUring["io_uring Subsystem"]:::os
        FileSys["File System"]:::os
    end

    %% Auxiliary Tools
    subgraph "Auxiliary"
        Docs["Architecture Docs (docs/architecture.md)"]:::aux
        Bench["Benchmark Harness (benchmarks/)"]:::aux
        Scripts["Visualization Scripts (scripts/)"]:::aux
        Tests["Test Suite (tests/)"]:::aux
        CI["Build & CI (Makefile, .github/workflows/)"]:::aux
    end

    %% Flow: Client to API
    Client -->|"calls"| API
    API --> ModExports
    ModExports --> Config
    ModExports --> Paths

    %% Write Path
    Paths -->|select topic| Allocator
    Allocator -->|"alloc memory"| WriterCore
    WriterCore --> BatchWrite
    BatchWrite --> StorageBackend
    StorageBackend --> FileSys
    FileSys --> WalFile
    WalFile --> IndexMgr
    IndexMgr --> Background
    Background --> FileSys

    %% Read Path
    Paths -->|select topic| ReaderCore
    ReaderCore --> BatchRead
    BatchRead --> IndexMgr
    IndexMgr --> StorageBackend
    StorageBackend --> FileSys
    FileSys --> ReaderCore
    ReaderCore -->|return entries| Client

    %% Batch Orchestrator
    BatchOrch --> WriterCore
    BatchOrch --> ReaderCore

    %% OS Integrations
    StorageBackend --> IOUring
    StorageBackend --> FileSys

    %% Auxiliary connections
    API -.-> Docs
    API -.-> Bench
    API -.-> Scripts
    API -.-> Tests
    API -.-> CI

    %% Click Events
    click API "https://github.com/nubskr/walrus/blob/master/src/lib.rs"
    click ModExports "https://github.com/nubskr/walrus/blob/master/src/wal/mod.rs"
    click Config "https://github.com/nubskr/walrus/blob/master/src/wal/config.rs"
    click Paths "https://github.com/nubskr/walrus/blob/master/src/wal/paths.rs"
    click BlockMgr "https://github.com/nubskr/walrus/blob/master/src/wal/block.rs"
    click StorageBackend "https://github.com/nubskr/walrus/blob/master/src/wal/storage.rs"
    click BatchOrch "https://github.com/nubskr/walrus/blob/master/src/wal/runtime/walrus.rs"
    click WriterCore "https://github.com/nubskr/walrus/blob/master/src/wal/runtime/writer.rs"
    click BatchWrite "https://github.com/nubskr/walrus/blob/master/src/wal/runtime/walrus_write.rs"
    click ReaderCore "https://github.com/nubskr/walrus/blob/master/src/wal/runtime/reader.rs"
    click BatchRead "https://github.com/nubskr/walrus/blob/master/src/wal/runtime/walrus_read.rs"
    click Allocator "https://github.com/nubskr/walrus/blob/master/src/wal/runtime/allocator.rs"
    click Background "https://github.com/nubskr/walrus/blob/master/src/wal/runtime/background.rs"
    click IndexMgr "https://github.com/nubskr/walrus/blob/master/src/wal/runtime/index.rs"
    click Docs "https://github.com/nubskr/walrus/blob/master/docs/architecture.md"
    click Bench "https://github.com/nubskr/walrus/tree/master/benchmarks/"
    click Scripts "https://github.com/nubskr/walrus/tree/master/scripts/"
    click Tests "https://github.com/nubskr/walrus/tree/master/tests/"
    click CI "https://github.com/nubskr/walrus/tree/master/Makefile"

    %% Styles
    classDef api fill:#D0E6FF,stroke:#333,stroke-width:1px
    classDef runtime fill:#DFFFD0,stroke:#333,stroke-width:1px
    classDef storage fill:#FFE4B5,stroke:#333,stroke-width:1px
    classDef os fill:#E0E0E0,stroke:#333,stroke-width:1px
    classDef aux fill:none,stroke:#999,stroke-dasharray:5,5
    classDef external fill:#F0F0F0,stroke:#333,stroke-width:1px
```
