dvorobiev/s3flood

Purpose and entry points

s3flood is a Python console tool for generating file datasets, load-testing S3-compatible storage, browsing buckets, and emitting live/final performance results. Primary invocation routes are module startup and CLI dispatch in src/s3flood/__main__.py, src/s3flood/cli.py, with a packaging/standalone entry wrapper in s3flood_entry.py.

Command and interaction layer

The CLI exposes dataset creation, benchmark runs, bucket browsing, and an interactive menu. Interactive flows coordinate config selection/editing, validation, destructive bucket actions, and launch the same core run/browser capabilities rather than a separate backend.

Paths: src/s3flood/cli.py, src/s3flood/interactive.py, src/s3flood/config_editor.py.

Configuration and application-scoped state

Run settings are loaded from YAML then overridden by CLI inputs; defaults and validation define profiles, endpoints, retry/queue/pattern controls, reporting paths, and client settings. Dataset location is application-level state (.s3flood.yml) rather than a durable per-run config value, with a command-line override.

Paths: src/s3flood/config.py, src/s3flood/defaults.py, src/s3flood/app_settings.py.

Dataset generation and operation source

Dataset creation produces grouped small/medium/large files with configurable total size and fill mode. During a run, the dataset module enumerates and orders local files; these become write inputs, while read/mixed profiles target corresponding remote object names. This is the boundary between local filesystem content and load scheduling.

Paths: src/s3flood/dataset.py, src/s3flood/runner.py.

Load orchestration and execution

The runner builds the selected profile (write, read, mixed), chooses endpoints (including round-robin/random cluster routing), applies order, warmup, burst behavior, bounded queues, and infinite looping. The executor performs concurrent operations, retries failed work with exponential backoff, and tracks per-operation endpoint/thread/attempt details. Endpoint affinity preserves the write endpoint for later reads.

Paths: src/s3flood/runner.py, src/s3flood/executor.py, src/s3flood/config.py.

S3 client and external-service boundary

S3 I/O is delegated through supported command-line clients—default AWS CLI, plus rclone or s3cmd—rather than an in-process SDK. The tool targets any S3-compatible endpoint, such as MinIO or AWS S3; AWS CLI configuration/environment overrides support credentials, profiles, multipart behavior, and separate process startup overhead measurement.

Paths: src/s3flood/executor.py, src/s3flood/config.py, src/s3flood/s3browser_io.py.

Observability and output pipeline

Each operation yields raw metric records for CSV and feeds aggregate latency, throughput, error, and timeline calculations. The dashboard renders live terminal progress/sparklines when interactive; non-TTY operation prints periodic summaries. Completion generates structured JSON reporting plus console summaries, excluding warmup from aggregate results while retaining raw data.

Paths: src/s3flood/metrics.py, src/s3flood/dashboard.py, src/s3flood/runner.py.

Two-panel bucket browser

A terminal Norton-Commander-style browser presents local filesystem and S3 bucket panels, including bucket/prefix traversal, object version inspection, cross-panel copy/move/delete, selection, cancellation, and progress dialogs. Browser-specific S3 calls are separated from terminal rendering/control, forming a distinct interactive management path from the load executor.

Paths: src/s3flood/browser.py, src/s3flood/s3browser_io.py, src/s3flood/dashboard.py.