# peak/s5cmd architecture

> Architecture diagram of the GitHub repository peak/s5cmd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/peak/s5cmd
- Repository: https://github.com/peak/s5cmd

## Overview

##### Main Components Identification

The system can be broken down into these major components:

a. Command Layer:
- Located in `/command` directory
- Handles all S3 operations (ls, cp, mv, rm, etc.)
- Implements command parsing and execution logic

b. Storage Layer:
- Located in `/storage` directory
- Abstracts storage operations for both S3 and local filesystem
- Implements S3 client and filesystem operations

c. Parallel Execution Engine:
- Located in `/parallel` directory
- Manages worker pools and concurrent operations
- Handles file descriptor limits

d. Supporting Components:
- Progress Bar (/progressbar)
- Logging System (/log)
- Ordered Writer (/orderedwriter)

**┌─────────────────────────────────────────────────────────────┐**
- Client Interface                        │
- (CLI Commands: ls, cp, mv, rm, sync, select, etc.)        │
- ┬─────────────────────────┬──────────────┘

**┌───────────────────▼─────┐       ┌──────────▼──────────┐**
- Command Processor     │       │   Command Runner    │
- (Parameter Validation,  │◄────►│(Batch Processing,   │
- Wildcard Expansion)    │       │  Parallel Execution) │
- ┬─────┘       └──────────┬──────────┘

**┌───────────────────▼────────────────────────▼──────────────┐**
- Parallel Execution Engine                  │
- (Worker Pools, Concurrent Operations)            │
- ┬────────────────────────┬──────────────┘

- ┌───────────▼──────────┐   ┌────────▼─────────┐
- Storage Layer     │   │  Progress/Logging │
- (S3 & Filesystem)   │   │     System       │
- ┬──────────┘   └──────────────────┘

- ┌───────────▼──────────┐
- External Systems   │
- (S3, Local Files)   │
- ┘

##### Key Interactions to Highlight

a. Command Flow:
- CLI input → Command Processor → Storage Layer
- Batch commands → Command Runner → Parallel Execution

b. Data Flow:
- Storage Layer ↔ External Systems
- Progress updates → Progress/Logging System

##### Important Details

a. Concurrency Model:
- Worker pool size (default 256)
- Parallel upload/download capabilities
- File descriptor management

b. Storage Abstractions:
- S3 client configuration
- Local filesystem operations
- URL handling

c. Error Handling:
- Retry logic
- Error reporting
- Integrity verification

##### Additional Notes

- Annotations for key configuration parameters
- Performance characteristics (worker pools, concurrent operations)
- Integration points with external systems

##### Implementation Details

- The modular architecture allowing for different storage backends
- The parallel execution model that enables high performance
- The separation of concerns between command processing and execution
- The extensible nature of the command system

This system design captures s5cmd's high-performance characteristics while maintaining clarity about how the components interact to achieve its functionality.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph CLI["Client Interface Layer"]
        Commands["CLI Commands<br/>(ls, cp, mv, rm, sync, etc.)"]
    end

    subgraph CmdLayer["Command Processing Layer"]
        CmdProc["Command Processor<br/>(Parameter Validation,<br/>Wildcard Expansion)"]
        CmdRun["Command Runner<br/>(Batch Processing)"]
        IndividualCmds["Individual Commands<br/>(cp, mv, ls, rm, sync)"]
    end

    subgraph ParallelLayer["Parallel Execution Layer"]
        ParallelEngine["Parallel Execution Engine<br/>(Worker Pools)"]
        GlobalPool["Global Worker Pool"]
        FDLimit["FD Limit Handler"]
    end

    subgraph StorageLayer["Storage Layer"]
        StorageInterface["Storage Interface"]
        S3Storage["S3 Storage"]
        FSStorage["Filesystem Storage"]
        URLHandler["URL Handler"]
    end

    subgraph SupportLayer["Supporting Systems"]
        ProgressBar["Progress Bar"]
        LogSystem["Logging System"]
        OrderedWriter["Ordered Writer"]
        ErrorHandler["Error Handler"]
    end

    subgraph External["External Systems"]
        S3["Amazon S3"]
        LocalFS["Local Filesystem"]
    end

    Commands --> CmdProc
    CmdProc --> CmdRun
    CmdRun --> IndividualCmds
    IndividualCmds --> ParallelEngine
    ParallelEngine --> GlobalPool
    ParallelEngine --> FDLimit
    ParallelEngine --> StorageInterface
    StorageInterface --> S3Storage
    StorageInterface --> FSStorage
    StorageInterface --> URLHandler
    S3Storage --> S3
    FSStorage --> LocalFS
    ParallelEngine --> ProgressBar
    ParallelEngine --> LogSystem
    ParallelEngine --> OrderedWriter
    ParallelEngine --> ErrorHandler

    %% Click events for component mapping
    click CmdProc "https://github.com/peak/s5cmd/blob/master/command/run.go"
    click IndividualCmds "https://github.com/peak/s5cmd/tree/master/command"
    click ParallelEngine "https://github.com/peak/s5cmd/blob/master/parallel/parallel.go"
    click GlobalPool "https://github.com/peak/s5cmd/blob/master/parallel/global.go"
    click FDLimit "https://github.com/peak/s5cmd/tree/master/parallel/fdlimit"
    click StorageInterface "https://github.com/peak/s5cmd/blob/master/storage/storage.go"
    click S3Storage "https://github.com/peak/s5cmd/blob/master/storage/s3.go"
    click FSStorage "https://github.com/peak/s5cmd/blob/master/storage/fs.go"
    click URLHandler "https://github.com/peak/s5cmd/blob/master/storage/url/url.go"
    click ProgressBar "https://github.com/peak/s5cmd/blob/master/progressbar/progressbar.go"
    click LogSystem "https://github.com/peak/s5cmd/blob/master/log/log.go"
    click OrderedWriter "https://github.com/peak/s5cmd/blob/master/orderedwriter/orderedwriter.go"
    click ErrorHandler "https://github.com/peak/s5cmd/blob/master/error/error.go"

    classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
    classDef cli fill:#e1f5fe,stroke:#01579b;
    classDef cmd fill:#e8f5e9,stroke:#1b5e20;
    classDef parallel fill:#fff3e0,stroke:#e65100;
    classDef storage fill:#f3e5f5,stroke:#4a148c;
    classDef support fill:#fff8e1,stroke:#f57f17;
    classDef external fill:#fbe9e7,stroke:#bf360c;

    class Commands cli;
    class CmdProc,CmdRun,IndividualCmds cmd;
    class ParallelEngine,GlobalPool,FDLimit parallel;
    class StorageInterface,S3Storage,FSStorage,URLHandler storage;
    class ProgressBar,LogSystem,OrderedWriter,ErrorHandler support;
    class S3,LocalFS external;
```
