How skyllc-ai/ultrafastfilesearch works
1. What this repository is
UFFS is a multi-crate Rust workspace for NTFS file search. The architecture is centered on one engine that can read NTFS metadata directly, build a compact index, keep that index resident in a daemon, and expose the same search capability through CLI, MCP, and other clients.
2. Top-level layering
The workspace is split into a few clear layers:
- low-level NTFS/MFT ingestion and indexing
- reusable search/query engine
- long-lived daemon and cache management
- user-facing clients and protocols
- formatting/output and utilities
This separation is important because most interfaces are thin wrappers over the same core search path.
3. Ingestion boundary: uffs-mft
uffs-mft is the lowest-level subsystem. It reads NTFS Master File Table data, parses records, resolves NTFS-specific structures, and builds the compact index representation. It includes platform-specific code for Windows live volume access, IOCP-based reading, parallel/pipelined readers, and offline MFT processing from captured files. This crate is the main boundary between raw filesystem bytes and the searchable in-memory model.
4. Index build flow
The build path is roughly: NTFS volume or MFT capture → sequential/parallel read buffers → record parsing and fixup → path/attribute reconstruction → compact index assembly → persisted cache. The code tree shows separate modules for parsing, readers, storage, path resolution, USN handling, and tree/index modeling, which indicates that record ingestion, logical reconstruction, and persistence are intentionally decoupled.
5. Search/query core: uffs-core
uffs-core is the shared engine used by higher-level consumers. It contains the query model, filters, pattern matching, glob/regex-like compilation, search dispatch, sorting, aggregation, and output shaping. This is where a user query becomes an executable plan against the index, and where result sets are filtered, ranked, paginated, or aggregated.
6. Accelerators and search mechanics
The core search path is optimized around data structures like tries, trigrams, bloom filters, path resolvers, and compact storage helpers. Those subsystems exist to make exact/prefix/substring/path queries fast without walking the filesystem. Aggregation and export are treated as first-class paths, not afterthoughts, which matters for diagramming because bulk result streaming is a separate branch from interactive search.
7. Long-lived serving layer: uffs-daemon
uffs-daemon is the runtime server that owns loaded indexes and answers requests over local IPC. It manages startup/shutdown, lifecycle, event handling, telemetry, and request handlers. The daemon is the main stateful boundary in the system: clients do not talk to NTFS directly; they talk to the daemon, which talks to the index/cache and optionally promotes or demotes drive state.
8. Cache and memory tiering
The daemon’s cache subsystem is a major architectural component. It persists compact drive indexes, reloads them quickly, and manages memory tiers such as hot, warm, parked, and cold. The cache tree includes background IO, journal loops, shard/registry management, prefetching, pressure handling, and cursor/state storage. For an architecture diagram, this should be shown as a stateful cache layer between disk persistence and in-memory search availability.
9. Client surface: uffs-cli, uffs-mcp, and uffs-client
uffs-cli is the main human-facing interface and command dispatcher. It parses args, maps commands into search/daemon/status operations, and formats results. uffs-mcp exposes the same capabilities through the Model Context Protocol for agent integrations, with tool handlers for search, drives, facets, status, and aggregation. uffs-client is the shared IPC library that encapsulates connection logic, daemon spawning, keepalive, sync behavior, and protocol types.
10. Result formatting and presentation
uffs-format is a separate formatting crate for rows, columns, attributes, datetimes, and output writers. This keeps presentation concerns isolated from query execution. The output path likely looks like: core search results → formatting layer → CLI stdout / CSV / structured MCP response. That separation is useful because the same search results must be rendered in multiple shapes without changing the engine.
11. Platform and deployment boundaries
Live NTFS reading is Windows-specific and requires elevated access, while offline analysis can run on macOS/Linux against saved MFT data and cached indexes. The repository therefore has both Windows runtime code and cross-platform offline paths. There are also IPC bridge modules for Windows/Unix differences, which shows that process communication is abstracted but still platform-aware.
12. Diagnostics, forensic tooling, and validation
uffs-diag, the many test suites, and the scripts under scripts/ support parity checks, benchmarks, Windows validation, and forensic analysis of MFT behavior. These tools are not part of the main runtime path, but they are important architecture support: they validate correctness across parsing, indexing, cache behavior, and cross-tool comparisons.
13. Build, release, and governance infrastructure
The repo includes substantial CI/release automation, supply-chain auditing, codegen helpers, and packaging scripts. Those are not runtime subsystems, but they do affect the architecture by enforcing build reproducibility, platform packaging, dependency policy, and release flow across the workspace.
14. How to draw the main data flow
The central diagram should show: NTFS volume or offline MFT file → uffs-mft ingestion/parsing → compact index/cache on disk and in memory → uffs-daemon serving queries via IPC → uffs-client-backed interfaces (uffs-cli, uffs-mcp) → formatted output. A secondary path should show cache promotion/demotion and reload from persisted state, because the daemon’s tiering model is a core part of the system behavior.
15. What not to flatten
Don’t draw this as a single search binary. The repository is architected as a reusable engine plus multiple interfaces, with explicit boundaries between filesystem ingestion, query execution, serving/runtime state, and presentation. The daemon and cache are not optional details; they are the key architectural elements that make the engine fast after the first load.
Main entry points: search Search exec · index Index mgr · src Client IPC · output Output shape
Components 21
Ingestion
NTFS reading, parsing, and compact index construction
- IO readers · Sequential, parallel, pipelined, and IOCP-backed MFT/volume readers
crates/uffs-mft/src/io - NTFS parse · Boot sector, records, metadata, fixup, and NTFS-specific structures
crates/uffs-mft/src/ntfs - Record parse · Reconstructs attributes, names, paths, and forensic views from raw records
crates/uffs-mft/src/parse - Index build · Builds the compact tree/index model and persists it for later reload
crates/uffs-mft/src/index - Load paths · Offline and live-drive load modes, cache reload, persistence, and USN application
crates/uffs-mft/src/reader
Search Engine
Shared query, matching, and result shaping
- Query model · Parses query intent into filters, matching rules, and operations
crates/uffs-core/src/query - Patterns · Glob and pattern parsing/compilation used by search and filters
crates/uffs-core/src/compiled_pattern - Search exec · Dispatches searches over compact storage with sorting, top-N, and field projection
crates/uffs-core/src/search - Aggregation · Grouping, rollups, pagination, export, and duplicate handling
crates/uffs-core/src/aggregate - Compact store · Compact loader, mmap, cache, bloom/trigram helpers, and path resolution
crates/uffs-core/src/compact_storage - Output shape · Column layout, display rows, and output bridge for consumers
crates/uffs-core/src/output
Daemon
Stateful server and index/cache lifecycle
- Startup · Boot, shutdown, lifecycle, logging, telemetry, and runtime orchestration
crates/uffs-daemon/src/startup.rs - Index mgr · Drive loading, refresh, hotload, search, stats, and tiering transitions
crates/uffs-daemon/src/index - Cache tiering · Persistent cache, shard registry, background IO, pressure, prefetch, and journal loops
crates/uffs-daemon/src/cache - IPC server · Local IPC, protocol handling, handlers, and Windows/Unix bridge
crates/uffs-daemon/src/ipc
Clients
User-facing and protocol-facing entrypoints
- Client IPC · Connection, daemon spawn, keepalive, sync/autostart, and shared protocol types
crates/uffs-client/src - CLI · Argument parsing, command dispatch, search, daemon management, and stdout interaction
crates/uffs-cli/src - MCP server · MCP tools, resources, prompts, and HTTP/process entrypoints
crates/uffs-mcp/src
Support
Formatting, diagnostics, security, and shared helpers
- Formatting · Rows, columns, attributes, datetimes, and writer helpers for output rendering
crates/uffs-format/src - Diagnostics · Parity checks, MFT analysis, and validation utilities for engine behavior
crates/uffs-diag/src - Security · Keystore, crypto, filesystem, pipe, and runtime directory helpers
crates/uffs-security/src
Connections 27
- IO readers → NTFS parse: raw records · Reads NTFS bytes and feeds parsed record structures
- NTFS parse → Record parse: parsed input · NTFS metadata is normalized into attribute/path reconstruction
- Record parse → Index build: index rows · Parsed records become compact index and tree models
- IO readers → Load paths: load modes · Live, offline, parallel, and capture-based read paths are coordinated here
- Load paths → Index build: persist/load · Builds or reloads compact index state from disk cache
- Load paths → Cache tiering: index cache · Daemon cache stores and restores the compact drive state
- Index build → Compact store: compact data · The searchable index is consumed by the shared storage/search layer
- Query model → Patterns: compile · Query intent is turned into executable matching patterns
- Query model → Search exec: plan · Filters and operations are dispatched into search execution
- Patterns → Search exec: match rules · Compiled patterns power exact/prefix/substring/path matching
- Compact store → Search exec: scan data · Compact storage provides the searchable backing structures
- Search exec → Aggregation: aggregate · Search results can branch into grouping, rollup, pagination, and export
- Search exec → Output shape: shape rows · Search results are transformed for display-oriented consumers
- Aggregation → Output shape: export view · Aggregated results are emitted through the same presentation layer
- Startup → Index mgr: serve index · Runtime startup hands off to the index manager
- Index mgr → Cache tiering: tier state · Index manager uses the cache layer for hot/warm/parked/cold state
- Index mgr → Search exec: queries · Daemon search handlers invoke the shared engine
- Index mgr → Aggregation: aggregates · Daemon exposes aggregation workflows over the shared engine
- IPC server → Index mgr: requests · IPC handlers route client commands into index management
- Client IPC → IPC server: ipc · Client library connects to the daemon transport
- CLI → Client IPC: runtime client · CLI uses the shared IPC client for daemon-backed operations
- CLI → Output shape: stdout · Command output is rendered through the shared formatting path
- MCP server → Client IPC: protocol client · MCP surfaces reuse the same daemon connection layer
- MCP server → Output shape: responses · Tool results are shaped for structured protocol output
- Diagnostics → Record parse: validate · Diagnostics compare parsing and record reconstruction behavior
- Diagnostics → Search exec: parity · Validation tools compare engine behavior across scenarios
- Security → Client IPC: runtime support · Security and runtime-directory helpers support client/daemon IPC setup