# skyllc-ai/ultrafastfilesearch architecture

> Architecture diagram of the GitHub repository skyllc-ai/ultrafastfilesearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skyllc-ai/ultrafastfilesearch
- Repository: https://github.com/skyllc-ai/ultrafastfilesearch

## Overview

1. **What this repository is**
   UFFS is a multi-crate Rust workspace for NTFS file search. The architecture is centered on one engine that can read NTFS metadata directly, build a compact index, keep that index resident in a daemon, and expose the same search capability through CLI, MCP, and other clients.

2. **Top-level layering**
   The workspace is split into a few clear layers:
   - low-level NTFS/MFT ingestion and indexing
   - reusable search/query engine
   - long-lived daemon and cache management
   - user-facing clients and protocols
   - formatting/output and utilities
   This separation is important because most interfaces are thin wrappers over the same core search path.

3. **Ingestion boundary: `uffs-mft`**
   `uffs-mft` is the lowest-level subsystem. It reads NTFS Master File Table data, parses records, resolves NTFS-specific structures, and builds the compact index representation. It includes platform-specific code for Windows live volume access, IOCP-based reading, parallel/pipelined readers, and offline MFT processing from captured files. This crate is the main boundary between raw filesystem bytes and the searchable in-memory model.

4. **Index build flow**
   The build path is roughly: NTFS volume or MFT capture → sequential/parallel read buffers → record parsing and fixup → path/attribute reconstruction → compact index assembly → persisted cache. The code tree shows separate modules for parsing, readers, storage, path resolution, USN handling, and tree/index modeling, which indicates that record ingestion, logical reconstruction, and persistence are intentionally decoupled.

5. **Search/query core: `uffs-core`**
   `uffs-core` is the shared engine used by higher-level consumers. It contains the query model, filters, pattern matching, glob/regex-like compilation, search dispatch, sorting, aggregation, and output shaping. This is where a user query becomes an executable plan against the index, and where result sets are filtered, ranked, paginated, or aggregated.

6. **Accelerators and search mechanics**
   The core search path is optimized around data structures like tries, trigrams, bloom filters, path resolvers, and compact storage helpers. Those subsystems exist to make exact/prefix/substring/path queries fast without walking the filesystem. Aggregation and export are treated as first-class paths, not afterthoughts, which matters for diagramming because bulk result streaming is a separate branch from interactive search.

7. **Long-lived serving layer: `uffs-daemon`**
   `uffs-daemon` is the runtime server that owns loaded indexes and answers requests over local IPC. It manages startup/shutdown, lifecycle, event handling, telemetry, and request handlers. The daemon is the main stateful boundary in the system: clients do not talk to NTFS directly; they talk to the daemon, which talks to the index/cache and optionally promotes or demotes drive state.

8. **Cache and memory tiering**
   The daemon’s cache subsystem is a major architectural component. It persists compact drive indexes, reloads them quickly, and manages memory tiers such as hot, warm, parked, and cold. The cache tree includes background IO, journal loops, shard/registry management, prefetching, pressure handling, and cursor/state storage. For an architecture diagram, this should be shown as a stateful cache layer between disk persistence and in-memory search availability.

9. **Client surface: `uffs-cli`, `uffs-mcp`, and `uffs-client`**
   `uffs-cli` is the main human-facing interface and command dispatcher. It parses args, maps commands into search/daemon/status operations, and formats results. `uffs-mcp` exposes the same capabilities through the Model Context Protocol for agent integrations, with tool handlers for search, drives, facets, status, and aggregation. `uffs-client` is the shared IPC library that encapsulates connection logic, daemon spawning, keepalive, sync behavior, and protocol types.

10. **Result formatting and presentation**
    `uffs-format` is a separate formatting crate for rows, columns, attributes, datetimes, and output writers. This keeps presentation concerns isolated from query execution. The output path likely looks like: core search results → formatting layer → CLI stdout / CSV / structured MCP response. That separation is useful because the same search results must be rendered in multiple shapes without changing the engine.

11. **Platform and deployment boundaries**
    Live NTFS reading is Windows-specific and requires elevated access, while offline analysis can run on macOS/Linux against saved MFT data and cached indexes. The repository therefore has both Windows runtime code and cross-platform offline paths. There are also IPC bridge modules for Windows/Unix differences, which shows that process communication is abstracted but still platform-aware.

12. **Diagnostics, forensic tooling, and validation**
    `uffs-diag`, the many test suites, and the scripts under `scripts/` support parity checks, benchmarks, Windows validation, and forensic analysis of MFT behavior. These tools are not part of the main runtime path, but they are important architecture support: they validate correctness across parsing, indexing, cache behavior, and cross-tool comparisons.

13. **Build, release, and governance infrastructure**
    The repo includes substantial CI/release automation, supply-chain auditing, codegen helpers, and packaging scripts. Those are not runtime subsystems, but they do affect the architecture by enforcing build reproducibility, platform packaging, dependency policy, and release flow across the workspace.

14. **How to draw the main data flow**
    The central diagram should show: NTFS volume or offline MFT file → `uffs-mft` ingestion/parsing → compact index/cache on disk and in memory → `uffs-daemon` serving queries via IPC → `uffs-client`-backed interfaces (`uffs-cli`, `uffs-mcp`) → formatted output. A secondary path should show cache promotion/demotion and reload from persisted state, because the daemon’s tiering model is a core part of the system behavior.

15. **What not to flatten**
    Don’t draw this as a single search binary. The repository is architected as a reusable engine plus multiple interfaces, with explicit boundaries between filesystem ingestion, query execution, serving/runtime state, and presentation. The daemon and cache are not optional details; they are the key architectural elements that make the engine fast after the first load.

Main entry points: [`crates/uffs-core/src/search`](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/search) (Search exec), [`crates/uffs-daemon/src/index`](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-daemon/src/index) (Index mgr), [`crates/uffs-client/src`](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-client/src) (Client IPC), [`crates/uffs-core/src/output`](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/output) (Output shape)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion"]
  node_uffs_mft_io["IO readers<br/>mft ingest IO"]
  node_uffs_mft_ntfs["NTFS parse"]
  node_uffs_mft_parse["Record parse<br/>mft parsing"]
  node_uffs_mft_index[("Index build")]
  node_uffs_mft_reader["Load paths<br/>reader/persistence"]
end

subgraph group_engine["Search Engine"]
  node_uffs_core_query["Query model<br/>query/filter core"]
  node_uffs_core_pattern["Patterns<br/>match compilation"]
  node_uffs_core_search["Search exec<br/>search dispatcher"]
  node_uffs_core_accum["Aggregation<br/>aggregate pipeline"]
  node_uffs_core_store[("Compact store<br/>index storage")]
  node_uffs_core_output["Output shape<br/>result shaping"]
end

subgraph group_daemon["Daemon"]
  node_uffs_daemon_start["Startup<br/>daemon lifecycle<br/>[startup.rs]"]
  node_uffs_daemon_index["Index mgr<br/>daemon index"]
  node_uffs_daemon_cache[("Cache tiering<br/>stateful cache")]
  node_uffs_daemon_ipc["IPC server<br/>daemon transport"]
end

subgraph group_clients["Clients"]
  node_uffs_client["Client IPC<br/>shared client"]
  node_uffs_cli["CLI"]
  node_uffs_mcp["MCP server<br/>tool protocol"]
end

subgraph group_support["Support"]
  node_uffs_format["Formatting<br/>presentation"]
  node_uffs_diag["Diagnostics<br/>forensics tools"]
  node_uffs_security["Security"]
end

node_uffs_mft_io -->|"raw records"| node_uffs_mft_ntfs
node_uffs_mft_ntfs -->|"parsed input"| node_uffs_mft_parse
node_uffs_mft_parse -->|"index rows"| node_uffs_mft_index
node_uffs_mft_io -->|"load modes"| node_uffs_mft_reader
node_uffs_mft_reader -->|"persist/load"| node_uffs_mft_index
node_uffs_mft_reader -->|"index cache"| node_uffs_daemon_cache
node_uffs_mft_index -->|"compact data"| node_uffs_core_store
node_uffs_core_query -->|"compile"| node_uffs_core_pattern
node_uffs_core_query -->|"plan"| node_uffs_core_search
node_uffs_core_pattern -->|"match rules"| node_uffs_core_search
node_uffs_core_store -->|"scan data"| node_uffs_core_search
node_uffs_core_search -->|"aggregate"| node_uffs_core_accum
node_uffs_core_search -->|"shape rows"| node_uffs_core_output
node_uffs_core_accum -->|"export view"| node_uffs_core_output
node_uffs_daemon_start -->|"serve index"| node_uffs_daemon_index
node_uffs_daemon_index -->|"tier state"| node_uffs_daemon_cache
node_uffs_daemon_index -->|"queries"| node_uffs_core_search
node_uffs_daemon_index -->|"aggregates"| node_uffs_core_accum
node_uffs_daemon_ipc -->|"requests"| node_uffs_daemon_index
node_uffs_client -->|"ipc"| node_uffs_daemon_ipc
node_uffs_cli -->|"runtime client"| node_uffs_client
node_uffs_cli -->|"stdout"| node_uffs_core_output
node_uffs_mcp -->|"protocol client"| node_uffs_client
node_uffs_mcp -->|"responses"| node_uffs_core_output
node_uffs_diag -.->|"validate"| node_uffs_mft_parse
node_uffs_diag -.->|"parity"| node_uffs_core_search
node_uffs_security -.->|"runtime support"| node_uffs_client

click node_uffs_mft_io "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/io"
click node_uffs_mft_ntfs "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/ntfs"
click node_uffs_mft_parse "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/parse"
click node_uffs_mft_index "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/index"
click node_uffs_mft_reader "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/reader"
click node_uffs_core_query "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/query"
click node_uffs_core_pattern "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/compiled_pattern"
click node_uffs_core_search "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/search"
click node_uffs_core_accum "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/aggregate"
click node_uffs_core_store "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/compact_storage"
click node_uffs_core_output "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/output"
click node_uffs_daemon_start "https://github.com/skyllc-ai/ultrafastfilesearch/blob/main/crates/uffs-daemon/src/startup.rs"
click node_uffs_daemon_index "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-daemon/src/index"
click node_uffs_daemon_cache "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-daemon/src/cache"
click node_uffs_daemon_ipc "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-daemon/src/ipc"
click node_uffs_client "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-client/src"
click node_uffs_cli "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-cli/src"
click node_uffs_mcp "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mcp/src"
click node_uffs_format "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-format/src"
click node_uffs_diag "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-diag/src"
click node_uffs_security "https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-security/src"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_uffs_mft_io,node_uffs_mft_ntfs,node_uffs_mft_parse,node_uffs_mft_index,node_uffs_mft_reader toneBlue
class node_uffs_core_query,node_uffs_core_pattern,node_uffs_core_search,node_uffs_core_accum,node_uffs_core_store,node_uffs_core_output toneAmber
class node_uffs_daemon_start,node_uffs_daemon_index,node_uffs_daemon_cache,node_uffs_daemon_ipc toneMint
class node_uffs_client,node_uffs_cli,node_uffs_mcp toneRose
class node_uffs_format,node_uffs_diag,node_uffs_security toneIndigo
```

## Components

### Ingestion

NTFS reading, parsing, and compact index construction

- [IO readers](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/io): mft ingest IO · Sequential, parallel, pipelined, and IOCP-backed MFT/volume readers · `crates/uffs-mft/src/io`
- [NTFS parse](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/ntfs): ntfs parser · Boot sector, records, metadata, fixup, and NTFS-specific structures · `crates/uffs-mft/src/ntfs`
- [Record parse](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/parse): mft parsing · Reconstructs attributes, names, paths, and forensic views from raw records · `crates/uffs-mft/src/parse`
- [Index build](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/index): compact index builder · Builds the compact tree/index model and persists it for later reload · `crates/uffs-mft/src/index`
- [Load paths](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mft/src/reader): reader/persistence · Offline and live-drive load modes, cache reload, persistence, and USN application · `crates/uffs-mft/src/reader`

### Search Engine

Shared query, matching, and result shaping

- [Query model](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/query): query/filter core · Parses query intent into filters, matching rules, and operations · `crates/uffs-core/src/query`
- [Patterns](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/compiled_pattern): match compilation · Glob and pattern parsing/compilation used by search and filters · `crates/uffs-core/src/compiled_pattern`
- [Search exec](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/search): search dispatcher · Dispatches searches over compact storage with sorting, top-N, and field projection · `crates/uffs-core/src/search`
- [Aggregation](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/aggregate): aggregate pipeline · Grouping, rollups, pagination, export, and duplicate handling · `crates/uffs-core/src/aggregate`
- [Compact store](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/compact_storage): index storage · Compact loader, mmap, cache, bloom/trigram helpers, and path resolution · `crates/uffs-core/src/compact_storage`
- [Output shape](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-core/src/output): result shaping · Column layout, display rows, and output bridge for consumers · `crates/uffs-core/src/output`

### Daemon

Stateful server and index/cache lifecycle

- [Startup](https://github.com/skyllc-ai/ultrafastfilesearch/blob/main/crates/uffs-daemon/src/startup.rs): daemon lifecycle · Boot, shutdown, lifecycle, logging, telemetry, and runtime orchestration · `crates/uffs-daemon/src/startup.rs`
- [Index mgr](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-daemon/src/index): daemon index · Drive loading, refresh, hotload, search, stats, and tiering transitions · `crates/uffs-daemon/src/index`
- [Cache tiering](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-daemon/src/cache): stateful cache · Persistent cache, shard registry, background IO, pressure, prefetch, and journal loops · `crates/uffs-daemon/src/cache`
- [IPC server](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-daemon/src/ipc): daemon transport · Local IPC, protocol handling, handlers, and Windows/Unix bridge · `crates/uffs-daemon/src/ipc`

### Clients

User-facing and protocol-facing entrypoints

- [Client IPC](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-client/src): shared client · Connection, daemon spawn, keepalive, sync/autostart, and shared protocol types · `crates/uffs-client/src`
- [CLI](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-cli/src): command client · Argument parsing, command dispatch, search, daemon management, and stdout interaction · `crates/uffs-cli/src`
- [MCP server](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-mcp/src): tool protocol · MCP tools, resources, prompts, and HTTP/process entrypoints · `crates/uffs-mcp/src`

### Support

Formatting, diagnostics, security, and shared helpers

- [Formatting](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-format/src): presentation · Rows, columns, attributes, datetimes, and writer helpers for output rendering · `crates/uffs-format/src`
- [Diagnostics](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-diag/src): forensics tools · Parity checks, MFT analysis, and validation utilities for engine behavior · `crates/uffs-diag/src`
- [Security](https://github.com/skyllc-ai/ultrafastfilesearch/tree/main/crates/uffs-security/src): runtime security · Keystore, crypto, filesystem, pipe, and runtime directory helpers · `crates/uffs-security/src`

## Connections

- IO readers → NTFS parse: raw records · Reads NTFS bytes and feeds parsed record structures
- NTFS parse → Record parse: parsed input · NTFS metadata is normalized into attribute/path reconstruction
- Record parse → Index build: index rows · Parsed records become compact index and tree models
- IO readers → Load paths: load modes · Live, offline, parallel, and capture-based read paths are coordinated here
- Load paths → Index build: persist/load · Builds or reloads compact index state from disk cache
- Load paths → Cache tiering: index cache · Daemon cache stores and restores the compact drive state
- Index build → Compact store: compact data · The searchable index is consumed by the shared storage/search layer
- Query model → Patterns: compile · Query intent is turned into executable matching patterns
- Query model → Search exec: plan · Filters and operations are dispatched into search execution
- Patterns → Search exec: match rules · Compiled patterns power exact/prefix/substring/path matching
- Compact store → Search exec: scan data · Compact storage provides the searchable backing structures
- Search exec → Aggregation: aggregate · Search results can branch into grouping, rollup, pagination, and export
- Search exec → Output shape: shape rows · Search results are transformed for display-oriented consumers
- Aggregation → Output shape: export view · Aggregated results are emitted through the same presentation layer
- Startup → Index mgr: serve index · Runtime startup hands off to the index manager
- Index mgr → Cache tiering: tier state · Index manager uses the cache layer for hot/warm/parked/cold state
- Index mgr → Search exec: queries · Daemon search handlers invoke the shared engine
- Index mgr → Aggregation: aggregates · Daemon exposes aggregation workflows over the shared engine
- IPC server → Index mgr: requests · IPC handlers route client commands into index management
- Client IPC → IPC server: ipc · Client library connects to the daemon transport
- CLI → Client IPC: runtime client · CLI uses the shared IPC client for daemon-backed operations
- CLI → Output shape: stdout · Command output is rendered through the shared formatting path
- MCP server → Client IPC: protocol client · MCP surfaces reuse the same daemon connection layer
- MCP server → Output shape: responses · Tool results are shaped for structured protocol output
- Diagnostics → Record parse: validate · Diagnostics compare parsing and record reconstruction behavior
- Diagnostics → Search exec: parity · Validation tools compare engine behavior across scenarios
- Security → Client IPC: runtime support · Security and runtime-directory helpers support client/daemon IPC setup
