# 5igi0/masstuffy architecture

> Architecture diagram of the GitHub repository 5igi0/masstuffy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/5igi0/masstuffy
- Repository: https://github.com/5igi0/masstuffy

## Overview

##### Project type and purpose
- Masstuffy is a Rust-based object-storage solution that exposes both:
- A command-line interface (CLI) for local management of collections and records.
- An HTTP server for remote management of collections and records.
- It uses SQLite (via SQLx) for metadata (CDX indexes) and WARC files on the filesystem for object payloads.

##### Main architectural components
a. CLI component (src/bin/cli.rs + cli_logic modules)
b. HTTP Server component (src/bin/server.rs + server_logic/endpoints)
c. Core library (src/lib.rs + modules: config, constants, database, filesystem, warc, utils)
d. Database layer (migrations/ + src/database)
e. WARC storage layer (src/filesystem + src/warc)
f. External interfaces (users calling CLI or sending HTTP requests)

##### Relationships and data flows
- The CLI and Server both sit “on top” of the same Core Library.
- Both use the Database module to run SQLx queries against the SQLite DB.
- Both use the Filesystem + WARC modules to read/write WARC files on disk.
- Server exposes HTTP endpoints (collections, dictionaries, record_getters, record_search) that map to server_logic handlers, which call Core Library functions.
- CLI logic modules map commands (create_collection, push_records, get_record, search) to Core Library calls.

##### Architectural patterns & design principles
- Layered architecture:
- Presentation layer: CLI & HTTP Server
- Application logic layer: server_logic & cli_logic modules
- Core services layer: config, constants, database, filesystem, warc, utils
- Infrastructure: SQLite via SQLx, migrations, raw files on disk.
- Shared library approach: Avoid duplication by centralizing business logic in src/lib.rs and its submodules.
- Migration-based database versioning (migrations/*.sql files).

##### Key technologies/frameworks
- Rust & Cargo (Cargo.toml)
- SQLx for async DB access (Cargo.toml + .sqlx files)
- SQLite as the metadata store (migrations folder)
- WARC file format for object storage
- Tokio async runtime (implied by async modules)
- CLI parsing (structopt or clap in cli.rs)
- HTTP framework (e.g. Axum or Warp in server.rs)

##### Final checklist
[ ] Annotations for config (env vars, config/mod.rs), utils (seek.rs)
[ ] Callouts for HTTP endpoints and CLI commands

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Actors
    Terminal["Terminal (CLI User)"]:::presentation
    HTTPClient["HTTP Client (curl/browser)"]:::presentation

    %% Presentation Layer
    subgraph "Presentation Layer"
        CLI["CLI Component"]:::presentation
        Server["HTTP Server Component"]:::presentation
    end

    %% CLI Logic Submodules
    subgraph "CLI Logic" 
        CreateColl["create_collection"]:::presentation
        GenDict["generate_dictionary"]:::presentation
        GetRecCLI["get_record"]:::presentation
        InitDB_CLI["init_db"]:::presentation
        InitFS_CLI["init_fs"]:::presentation
        PushRec["push_records"]:::presentation
        SearchCLI["search"]:::presentation
        CLI_Mod["cli_logic mod"]:::presentation
    end

    %% Server Logic Submodules
    subgraph "Server Logic"
        EndpointsMod["endpoints mod"]:::presentation
        CollectionsEP["collections endpoint"]:::presentation
        DictEP["dictionaries endpoint"]:::presentation
        RecGetEP["record_getters endpoint"]:::presentation
        RecSearchEP["record_search endpoint"]:::presentation
        Server_Mod["server_logic mod"]:::presentation
    end

    %% Core Library
    subgraph "Core Library" 
        Config["Config"]:::core
        Constants["Constants"]:::core
        DBMod["Database mod"]:::core
        DBStructs["Database structs"]:::core
        FSMod["Filesystem mod"]:::core
        FSCollections["collections store"]:::core
        FSDict["dictionary store"]:::core
        WARCMod["WARC mod"]:::core
        WARC_CDX["CDX index"]:::core
        WARC_MURL["massaged_url"]:::core
        UtilsMod["Utils mod"]:::core
        UtilsSeek["seek helper"]:::core
    end

    %% Infrastructure & Storage
    subgraph "Storage & Infra"
        SQLiteDB["SQLite DB"]:::storage
        WARCFiles["Disk WARC files"]:::storage
        Migrations["migrations/"]:::infra
        SQLxArt[".sqlx query artifacts"]:::infra
        CargoToml["Cargo.toml"]:::infra
        CargoLock["Cargo.lock"]:::infra
        README["README.md"]:::infra
        DocEP["doc/Endpoints.md"]:::infra
        DocEnv["doc/EnvironmentVariables.md"]:::infra
        DocFL["doc/FileLayout.md"]:::infra
        LICENSE["LICENSE.txt"]:::infra
        Gitignore[".gitignore"]:::infra
        SQLxMeta[".sqlx/"]:::infra
    end

    %% Relationships
    Terminal -->|"commands"| CLI
    HTTPClient -->|"HTTP requests"| Server

    CLI -->|"invokes"| CLI_Mod
    CLI_Mod -->|calls| Config
    CLI_Mod -->|calls| DBMod
    CLI_Mod -->|calls| FSMod
    CLI_Mod -->|calls| WARCMod
    CLI_Mod -->|calls| UtilsMod

    Server -->|"invokes"| Server_Mod
    Server_Mod -->|routes via Endpoints| EndpointsMod
    CollectionsEP -->|calls| Server_Mod
    DictEP -->|calls| Server_Mod
    RecGetEP -->|calls| Server_Mod
    RecSearchEP -->|calls| Server_Mod
    Server_Mod -->|calls| Config
    Server_Mod -->|calls| DBMod
    Server_Mod -->|calls| FSMod
    Server_Mod -->|calls| WARCMod
    Server_Mod -->|calls| UtilsMod

    DBMod -->|uses| DBStructs
    DBMod -->|"SQLx queries"| SQLiteDB
    FSMod -->|manages| FSCollections
    FSMod -->|manages| FSDict
    FSMod -->|"file read/write"| WARCFiles
    WARCMod -->|handles| WARC_CDX
    WARCMod -->|handles| WARC_MURL
    WARCMod -->|"indexing"| WARCFiles
    UtilsMod -->|provides| UtilsSeek

    Migrations -->|"schema migrations"| SQLiteDB
    SQLxArt -->|"query metadata"| DBMod
    CargoToml -->|defines deps| DBMod
    SQLxMeta -->|"metadata"| SQLxArt

    %% Click Events
    click CLI "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli.rs"
    click CreateColl "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli_logic/create_collection.rs"
    click GenDict "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli_logic/generate_dictionary.rs"
    click GetRecCLI "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli_logic/get_record.rs"
    click InitDB_CLI "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli_logic/init_db.rs"
    click InitFS_CLI "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli_logic/init_fs.rs"
    click PushRec "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli_logic/push_records.rs"
    click SearchCLI "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli_logic/search.rs"
    click CLI_Mod "https://github.com/5igi0/masstuffy/blob/master/src/bin/cli_logic/mod.rs"
    click Server "https://github.com/5igi0/masstuffy/blob/master/src/bin/server.rs"
    click EndpointsMod "https://github.com/5igi0/masstuffy/blob/master/src/bin/server_logic/endpoints/mod.rs"
    click CollectionsEP "https://github.com/5igi0/masstuffy/blob/master/src/bin/server_logic/endpoints/collections.rs"
    click DictEP "https://github.com/5igi0/masstuffy/blob/master/src/bin/server_logic/endpoints/dictionaries.rs"
    click RecGetEP "https://github.com/5igi0/masstuffy/blob/master/src/bin/server_logic/endpoints/record_getters.rs"
    click RecSearchEP "https://github.com/5igi0/masstuffy/blob/master/src/bin/server_logic/endpoints/record_search.rs"
    click Server_Mod "https://github.com/5igi0/masstuffy/blob/master/src/bin/server_logic/mod.rs"
    click Config "https://github.com/5igi0/masstuffy/blob/master/src/config/mod.rs"
    click Constants "https://github.com/5igi0/masstuffy/blob/master/src/constants.rs"
    click DBMod "https://github.com/5igi0/masstuffy/blob/master/src/database/mod.rs"
    click DBStructs "https://github.com/5igi0/masstuffy/blob/master/src/database/structs.rs"
    click FSMod "https://github.com/5igi0/masstuffy/blob/master/src/filesystem/mod.rs"
    click FSCollections "https://github.com/5igi0/masstuffy/blob/master/src/filesystem/collections.rs"
    click FSDict "https://github.com/5igi0/masstuffy/blob/master/src/filesystem/dict_store.rs"
    click WARCMod "https://github.com/5igi0/masstuffy/blob/master/src/warc/mod.rs"
    click WARC_CDX "https://github.com/5igi0/masstuffy/blob/master/src/warc/cdx.rs"
    click WARC_MURL "https://github.com/5igi0/masstuffy/blob/master/src/warc/massaged_url.rs"
    click UtilsMod "https://github.com/5igi0/masstuffy/blob/master/src/utils/mod.rs"
    click UtilsSeek "https://github.com/5igi0/masstuffy/blob/master/src/utils/seek.rs"
    click Migrations "https://github.com/5igi0/masstuffy/tree/master/migrations/"
    click SQLxArt "https://github.com/5igi0/masstuffy/tree/master/.sqlx/"
    click CargoToml "https://github.com/5igi0/masstuffy/blob/master/Cargo.toml"

    %% Styles
    classDef presentation fill:#D6EAF8,stroke:#1B4F72,stroke-width:1px
    classDef core fill:#D5F5E3,stroke:#196F3D,stroke-width:1px
    classDef storage fill:#F2F3F4,stroke:#707B7C,stroke-width:1px
    classDef infra fill:#FCF3CF,stroke:#9A7D0A,stroke-width:1px
```
