# axboe/fio architecture

> Architecture diagram of the GitHub repository axboe/fio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/axboe/fio
- Repository: https://github.com/axboe/fio

## Overview

##### Project type and purpose
- Fio is an open-source, command-line I/O workload generator and benchmark tool written in C. It reads “job files” or command-line parameters to spawn threads/processes that execute customizable I/O patterns across many storage APIs and back-ends. Its key value is flexibility and extensibility via pluggable I/O engines.

##### High-level file-structure analysis
- Top level: configure/build scripts (configure, Makefile), documentation (doc/, README.rst), CI definitions (.github/, ci/), examples/, tools/, unittests/
- Core C sources: fio.c (entrypoint), parser (parse.c, compiler/), job scheduler (flow.c, helper_thread.c), I/O engines (engines/), OS abstraction (os/, oslib/), architecture‐specific code (arch/), shared libraries/utilities (lib/)
- Extensibility points: each file in engines/ is a plugin implementing a struct of callbacks that the scheduler invokes; JSON, log, statistics modules for output; tools/ for post-processing and visualization.

##### Key README insights
- Fio spawns threads/processes per job, each inheriting global parameters
- Engines are compiled in or enabled by configure (libaio, HTTP, gfio GUI)
- Cross-platform support via os/ and oslib/ abstractions
- Documentation generated via Sphinx from doc/

##### Main components
a. CLI Front-end & Job Parser
- fio.c, parse.c, compiler/*: parse .fio jobfiles or CLI args into in-memory job descriptions
b. Job Scheduler & Control Plane
- flow.c, helper_thread.c, thread_options.h: spawn threads/processes, assign per-job runtimes, collect stats
c. I/O Engine Manager & Plugin Layer
- ioengines.c/ioengines.h: engine registry
- engines/*.c: individual back-ends (libaio, mmap, HTTP, POSIX AIO, GPU Direct RDMA, etc.)
d. OS Abstraction & Architecture Layer
- os/, oslib/: wrap platform syscalls (threads, file I/O, networking, timers)
- arch/: architecture-specific optimizations (e.g., CRC acceleration)
e. Utility Libraries
- lib/: common data structures (rbtree, bloom, rand), math utilities, memory align, logging, JSON output
f. Output & Reporting
- log.c/iolog.c: result logging, histogram generation
- tools/: post-processing scripts (fio2gnuplot, fiograph)
g. Testing & CI
- t/: unit and integration tests, fuzzers
- unittests/: C-level unit tests
- .github/ and ci/: automated build, QEMU VM integration, fuzz testing pipelines

##### Relationships and data flow
- User → CLI front-end → Job Parser → Job Descriptor objects
- Scheduler/Controller → spawns Worker threads/processes
- Each Worker → Engine Manager → Engine Plugin → OS Abstraction → Host Kernel/Storage Hardware
- Worker → gathers performance metrics → Controller → Aggregator → Logger/Output
- External tools consume output logs for visualization
- CI pipelines enforce build → test → fuzz → package

##### Architectural patterns & design principles
- Plugin architecture for I/O engines (dynamic registration via ioengines.c)
- Layered design: CLI/Parser → Scheduler → Engines → OS → Hardware
- Abstraction for portability (os/, oslib/)
- Separation of concerns: parsing, scheduling, I/O, logging, utilities
- Test harness isolated from production code (t/, unittests/)

##### Elements
- Clear labels for each component block and layer
- CLI → Parser → Scheduler → Worker → Engine → OS → Hardware
- Worker → Stats Aggregator → Logger → Tools
- Core engine (blue)
- Engine plugins (green)
- Platform abstractions (orange)
- Utilities & tools (gray)
- Tests/CI (purple)

## Diagram (Mermaid)

```mermaid
graph TB
    %% External entities
    User((User)):::external
    Repo((Source Repo)):::external

    %% Core Engine Layer
    subgraph "Core Engine" 
        direction TB
        FIOC["CLI Front-end (fio.c)"]:::core
        PARSE["Parser (parse.c)"]:::core
        COMP["Compiler (compiler/)"]:::core
        Flow["Scheduler (flow.c)"]:::core
        Helper["Helper Threads (helper_thread.c)"]:::core
        ThreadOpts["Thread Options (thread_options.h)"]:::core
        Worker["Worker Threads"]:::core
        Stats["Stats Aggregator"]:::core
    end

    %% I/O Engine Manager & Plugins
    subgraph "I/O Engine Layer"
        direction TB
        EngineMgr["Engine Manager (ioengines.c)"]:::core
        EngineHdr["Engine Manager Header (ioengines.h)"]:::core
        EnginesDir["Plugins (engines/)"]:::plugins
    end

    %% Platform Abstraction
    subgraph "Platform Abstraction"
        direction TB
        OS["OS Abstraction (os/)"]:::platform
        OSLib["OS Lib (oslib/)"]:::platform
        Arch["Architecture (arch/)"]:::platform
        Kernel((Host Kernel/Hardware)):::platform
    end

    %% Utilities & Tools
    subgraph "Utilities & Tools"
        direction TB
        Lib["Utility Libraries (lib/)"]:::utils
        Log["Logger (log.c)"]:::utils
        IOLog["IO Logger (iolog.c)"]:::utils
        Tools["Post-Processing Tools (tools/)"]:::utils
    end

    %% Testing & CI
    subgraph "Testing & CI"
        direction TB
        Tests["Integration Tests (t/)"]:::tests
        Unit["Unit Tests (unittests/)"]:::tests
        GH["GitHub Workflows (.github/)"]:::tests
        CI_Scripts["CI Scripts (ci/)"]:::tests
        Build["Build"]:::tests
        Package((Package)):::tests
    end

    %% Core flows
    User --> FIOC
    FIOC --> PARSE
    PARSE --> COMP
    COMP --> Flow
    COMP --> Helper
    COMP --> ThreadOpts
    Flow --> Worker
    Helper --> Worker
    ThreadOpts --> Worker
    Worker --> EngineMgr
    Worker --> Stats

    %% Engine layer flows
    EngineMgr --> EngineHdr
    EngineMgr --> EnginesDir
    EnginesDir --> OS
    OS --> Kernel
    OSLib --> Kernel
    Arch --> Kernel

    %% Stats & output
    Stats --> Log
    Stats --> IOLog
    Log --> Tools

    %% CI flows
    Repo --> Build
    Build --> Tests
    Build --> Unit
    Tests --> GH
    Unit --> GH
    GH --> CI_Scripts
    CI_Scripts --> Package

    %% Click Events
    click FIOC "https://github.com/axboe/fio/blob/master/fio.c"
    click PARSE "https://github.com/axboe/fio/blob/master/parse.c"
    click COMP "https://github.com/axboe/fio/tree/master/compiler/"
    click Flow "https://github.com/axboe/fio/blob/master/flow.c"
    click Helper "https://github.com/axboe/fio/blob/master/helper_thread.c"
    click ThreadOpts "https://github.com/axboe/fio/blob/master/thread_options.h"
    click EngineMgr "https://github.com/axboe/fio/blob/master/ioengines.c"
    click EngineHdr "https://github.com/axboe/fio/blob/master/ioengines.h"
    click EnginesDir "https://github.com/axboe/fio/tree/master/engines/"
    click OS "https://github.com/axboe/fio/tree/master/os/"
    click OSLib "https://github.com/axboe/fio/tree/master/oslib/"
    click Arch "https://github.com/axboe/fio/tree/master/arch/"
    click Lib "https://github.com/axboe/fio/tree/master/lib/"
    click Log "https://github.com/axboe/fio/blob/master/log.c"
    click IOLog "https://github.com/axboe/fio/blob/master/iolog.c"
    click Tools "https://github.com/axboe/fio/tree/master/tools/"
    click Tests "https://github.com/axboe/fio/tree/master/t/"
    click Unit "https://github.com/axboe/fio/tree/master/unittests/"
    click GH "https://github.com/axboe/fio/tree/master/.github/"
    click CI_Scripts "https://github.com/axboe/fio/tree/master/ci/"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef plugins fill:#90EE90,stroke:#333,stroke-width:1px
    classDef platform fill:#FFA500,stroke:#333,stroke-width:1px
    classDef utils fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef tests fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#333,stroke-width:1px
```
