# dvorobiev/s3flood architecture

> Architecture diagram of the GitHub repository dvorobiev/s3flood, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dvorobiev/s3flood
- Repository: https://github.com/dvorobiev/s3flood

## Overview

#### Purpose and entry points
`s3flood` is a Python console tool for generating file datasets, load-testing S3-compatible storage, browsing buckets, and emitting live/final performance results. Primary invocation routes are module startup and CLI dispatch in `src/s3flood/__main__.py`, `src/s3flood/cli.py`, with a packaging/standalone entry wrapper in `s3flood_entry.py`.

#### Command and interaction layer
The CLI exposes dataset creation, benchmark runs, bucket browsing, and an interactive menu. Interactive flows coordinate config selection/editing, validation, destructive bucket actions, and launch the same core run/browser capabilities rather than a separate backend.
Paths: `src/s3flood/cli.py`, `src/s3flood/interactive.py`, `src/s3flood/config_editor.py`.

#### Configuration and application-scoped state
Run settings are loaded from YAML then overridden by CLI inputs; defaults and validation define profiles, endpoints, retry/queue/pattern controls, reporting paths, and client settings. Dataset location is application-level state (`.s3flood.yml`) rather than a durable per-run config value, with a command-line override.
Paths: `src/s3flood/config.py`, `src/s3flood/defaults.py`, `src/s3flood/app_settings.py`.

#### Dataset generation and operation source
Dataset creation produces grouped small/medium/large files with configurable total size and fill mode. During a run, the dataset module enumerates and orders local files; these become write inputs, while read/mixed profiles target corresponding remote object names. This is the boundary between local filesystem content and load scheduling.
Paths: `src/s3flood/dataset.py`, `src/s3flood/runner.py`.

#### Load orchestration and execution
The runner builds the selected profile (`write`, `read`, `mixed`), chooses endpoints (including round-robin/random cluster routing), applies order, warmup, burst behavior, bounded queues, and infinite looping. The executor performs concurrent operations, retries failed work with exponential backoff, and tracks per-operation endpoint/thread/attempt details. Endpoint affinity preserves the write endpoint for later reads.
Paths: `src/s3flood/runner.py`, `src/s3flood/executor.py`, `src/s3flood/config.py`.

#### S3 client and external-service boundary
S3 I/O is delegated through supported command-line clients—default AWS CLI, plus rclone or s3cmd—rather than an in-process SDK. The tool targets any S3-compatible endpoint, such as MinIO or AWS S3; AWS CLI configuration/environment overrides support credentials, profiles, multipart behavior, and separate process startup overhead measurement.
Paths: `src/s3flood/executor.py`, `src/s3flood/config.py`, `src/s3flood/s3browser_io.py`.

#### Observability and output pipeline
Each operation yields raw metric records for CSV and feeds aggregate latency, throughput, error, and timeline calculations. The dashboard renders live terminal progress/sparklines when interactive; non-TTY operation prints periodic summaries. Completion generates structured JSON reporting plus console summaries, excluding warmup from aggregate results while retaining raw data.
Paths: `src/s3flood/metrics.py`, `src/s3flood/dashboard.py`, `src/s3flood/runner.py`.

#### Two-panel bucket browser
A terminal Norton-Commander-style browser presents local filesystem and S3 bucket panels, including bucket/prefix traversal, object version inspection, cross-panel copy/move/delete, selection, cancellation, and progress dialogs. Browser-specific S3 calls are separated from terminal rendering/control, forming a distinct interactive management path from the load executor.
Paths: `src/s3flood/browser.py`, `src/s3flood/s3browser_io.py`, `src/s3flood/dashboard.py`.

Main entry points: [`src/s3flood/cli.py`](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/cli.py) (CLI dispatch), [`src/s3flood/runner.py`](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/runner.py) (Run orchestrator), [`src/s3flood/interactive.py`](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/interactive.py) (Interactive menu), [`src/s3flood/config.py`](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/config.py) (Run configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry & interaction"]
  node_module_entry["Module startup<br/>Python entry point<br/>[__main__.py]"]
  node_standalone_entry["Standalone wrapper<br/>Packaging entry<br/>[s3flood_entry.py]"]
  node_cli["CLI dispatch<br/>Command interface<br/>[cli.py]"]
  node_interactive["Interactive menu<br/>Terminal workflow<br/>[interactive.py]"]
  node_config_editor["Config editor<br/>Terminal editor<br/>[config_editor.py]"]
end

subgraph group_setup["Configuration &amp; data"]
  node_config["Run configuration<br/>YAML settings<br/>[config.py]"]
  node_defaults["Defaults<br/>[defaults.py]"]
  node_app_settings["Dataset location state<br/>Application settings<br/>[app_settings.py]"]
  node_dataset["Dataset generator<br/>Local filesystem<br/>[dataset.py]"]
end

subgraph group_load["Load execution"]
  node_runner["Run orchestrator<br/>Benchmark scheduler<br/>[runner.py]"]
  node_executor["Concurrent executor<br/>Operation worker pool<br/>[executor.py]"]
  node_s3_clients{{"S3 command clients<br/>External CLI adapters<br/>[executor.py]"}}
  node_s3_service[("S3-compatible storage<br/>External service")]
end

subgraph group_output["Metrics &amp; output"]
  node_metrics["Metric aggregation<br/>Performance analytics<br/>[metrics.py]"]
  node_dashboard["Live dashboard<br/>Terminal reporting<br/>[dashboard.py]"]
  node_reports["Final reports<br/>CSV and JSON output<br/>[runner.py]"]
end

subgraph group_browser["Bucket browser"]
  node_browser["Two-panel browser<br/>Terminal file manager<br/>[browser.py]"]
  node_browser_io["Browser S3 I/O<br/>S3 browser adapter<br/>[s3browser_io.py]"]
end

node_module_entry -->|"starts"| node_cli
node_standalone_entry -->|"starts"| node_cli
node_cli -->|"opens"| node_interactive
node_interactive -->|"edits settings"| node_config_editor
node_cli -->|"loads and overrides"| node_config
node_interactive -->|"selects and validates"| node_config
node_config -->|"uses"| node_defaults
node_cli -->|"resolves dataset location"| node_app_settings
node_app_settings -->|"locates"| node_dataset
node_cli -->|"creates"| node_dataset
node_cli -->|"launches"| node_runner
node_interactive -->|"launches"| node_runner
node_config -->|"profiles and controls"| node_runner
node_dataset -->|"local file operations"| node_runner
node_runner -->|"schedules operations"| node_executor
node_executor -->|"invokes"| node_s3_clients
node_s3_clients -->|"S3 requests"| node_s3_service
node_executor -->|"raw operation records"| node_metrics
node_metrics -->|"live aggregates"| node_dashboard
node_metrics -->|"final aggregates"| node_reports
node_runner -->|"writes raw CSV and JSON"| node_reports
node_cli -->|"opens"| node_browser
node_interactive -->|"opens"| node_browser
node_browser -->|"storage actions"| node_browser_io
node_browser_io -->|"browser S3 calls"| node_s3_service
node_browser -->|"progress dialogs"| node_dashboard

click node_module_entry "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/__main__.py"
click node_standalone_entry "https://github.com/dvorobiev/s3flood/blob/main/s3flood_entry.py"
click node_cli "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/cli.py"
click node_interactive "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/interactive.py"
click node_config_editor "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/config_editor.py"
click node_config "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/config.py"
click node_defaults "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/defaults.py"
click node_app_settings "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/app_settings.py"
click node_dataset "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/dataset.py"
click node_runner "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/runner.py"
click node_executor "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/executor.py"
click node_s3_clients "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/executor.py"
click node_metrics "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/metrics.py"
click node_dashboard "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/dashboard.py"
click node_reports "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/runner.py"
click node_browser "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/browser.py"
click node_browser_io "https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/s3browser_io.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_module_entry,node_standalone_entry,node_cli,node_interactive,node_config_editor toneBlue
class node_config,node_defaults,node_app_settings,node_dataset toneAmber
class node_runner,node_executor,node_s3_clients,node_s3_service toneMint
class node_metrics,node_dashboard,node_reports toneRose
class node_browser,node_browser_io toneIndigo
```

## Components

### Entry & interaction

- [Module startup](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/__main__.py): Python entry point · `src/s3flood/__main__.py`
- [Standalone wrapper](https://github.com/dvorobiev/s3flood/blob/main/s3flood_entry.py): Packaging entry · `s3flood_entry.py`
- [CLI dispatch](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/cli.py): Command interface · Starts dataset, benchmark, browser, and interactive commands. · `src/s3flood/cli.py`
- [Interactive menu](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/interactive.py): Terminal workflow · Reuses run and browser capabilities. · `src/s3flood/interactive.py`
- [Config editor](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/config_editor.py): Terminal editor · `src/s3flood/config_editor.py`

### Configuration & data

- [Run configuration](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/config.py): YAML settings · Loads YAML, CLI overrides, profiles, and validation. · `src/s3flood/config.py`
- [Defaults](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/defaults.py): Configuration defaults · `src/s3flood/defaults.py`
- [Dataset location state](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/app_settings.py): Application settings · Stores dataset location in .s3flood.yml. · `src/s3flood/app_settings.py`
- [Dataset generator](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/dataset.py): Local filesystem · Creates grouped files and enumerates ordered write inputs. · `src/s3flood/dataset.py`

### Load execution

- [Run orchestrator](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/runner.py): Benchmark scheduler · Builds profiles and applies routing, ordering, warmup, bursts, queues, and looping. · `src/s3flood/runner.py`
- [Concurrent executor](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/executor.py): Operation worker pool · Runs operations, retries failures, and retains endpoint affinity. · `src/s3flood/executor.py`
- [S3 command clients](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/executor.py): External CLI adapters · AWS CLI, rclone, or s3cmd perform S3 I/O. · `src/s3flood/executor.py`
- **S3-compatible storage**: External service · Target may be MinIO, AWS S3, or another compatible endpoint.

### Metrics & output

- [Metric aggregation](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/metrics.py): Performance analytics · Calculates latency, throughput, errors, and timeline aggregates. · `src/s3flood/metrics.py`
- [Live dashboard](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/dashboard.py): Terminal reporting · Renders interactive progress or periodic non-TTY summaries. · `src/s3flood/dashboard.py`
- [Final reports](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/runner.py): CSV and JSON output · Keeps raw operation data and emits final summaries. · `src/s3flood/runner.py`

### Bucket browser

- [Two-panel browser](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/browser.py): Terminal file manager · Manages local files, buckets, prefixes, objects, and cross-panel actions. · `src/s3flood/browser.py`
- [Browser S3 I/O](https://github.com/dvorobiev/s3flood/blob/main/src/s3flood/s3browser_io.py): S3 browser adapter · Separates browser storage calls from terminal control. · `src/s3flood/s3browser_io.py`

## Connections

- Module startup → CLI dispatch: starts
- Standalone wrapper → CLI dispatch: starts
- CLI dispatch → Interactive menu: opens
- Interactive menu → Config editor: edits settings
- CLI dispatch → Run configuration: loads and overrides
- Interactive menu → Run configuration: selects and validates
- Run configuration → Defaults: uses
- CLI dispatch → Dataset location state: resolves dataset location
- Dataset location state → Dataset generator: locates
- CLI dispatch → Dataset generator: creates
- CLI dispatch → Run orchestrator: launches
- Interactive menu → Run orchestrator: launches
- Run configuration → Run orchestrator: profiles and controls
- Dataset generator → Run orchestrator: local file operations
- Run orchestrator → Concurrent executor: schedules operations
- Concurrent executor → S3 command clients: invokes
- S3 command clients → S3-compatible storage: S3 requests
- Concurrent executor → Metric aggregation: raw operation records
- Metric aggregation → Live dashboard: live aggregates
- Metric aggregation → Final reports: final aggregates
- Run orchestrator → Final reports: writes raw CSV and JSON
- CLI dispatch → Two-panel browser: opens
- Interactive menu → Two-panel browser: opens
- Two-panel browser → Browser S3 I/O: storage actions
- Browser S3 I/O → S3-compatible storage: browser S3 calls
- Two-panel browser → Live dashboard: progress dialogs