# sa7mon/s3scanner architecture

> Architecture diagram of the GitHub repository sa7mon/s3scanner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sa7mon/s3scanner
- Repository: https://github.com/sa7mon/s3scanner

## Overview

##### Project Type and Purpose
- S3Scanner is a command-line, multi-threaded scanning tool (written in Go) for discovering open S3 buckets across AWS and other cloud providers. It can ingest bucket names via CLI, file, or RabbitMQ, check permissions and optionally enumerate objects, then output results to stdout or persist them in a PostgreSQL database.

##### High-Level Architecture and Major Components
a. CLI Frontend
- Entry point (main.go + cmd/…) handling subcommands:
– s3scanner (bucket scan)
– regioncheck (validate provider regions)
– mqingest (publish buckets into RabbitMQ)
- Parses flags (`-bucket`, `-mq`, `-db`, `-provider`, `-threads`, etc.) and loads config.yml.
b. Scanner Core
- bucket package: represents bucket objects and metadata
- permission package: checks ACLs (read/write/ACP/full control)
- collection & groups: processes per-bucket results and groups findings
- enumeration logic (optional, single-threaded per bucket)
c. Provider Clients
- provider package: concrete implementations for AWS, GCP, DigitalOcean, Linode, DreamHost, Scaleway, Wasabi, and “custom.”
- clientmap: dynamic mapping of provider names to client constructors
- credentials & endpoint resolution via config.yml
d. Messaging Layer (RabbitMQ)
- mq package: connects to RabbitMQ using `mq.uri` and `mq.queue_name`
- Worker component (worker/mq_worker.go) consumes bucket names and pushes them into the Scanner Core
e. Persistence Layer (PostgreSQL via GORM)
- db package: opens a GORM connection, auto-migrates schema, and writes scan results
- Entities: bucket existence, region, permission details, enumeration results
f. Supporting Utilities
- log package: hooks and JSON formatter for structured logging
- config management: config.yml file resolution (current dir, /etc/s3scanner, $HOME/.s3scanner)
g. Dev & Packaging
- Docker Compose (.dev/docker-compose.yml) orchestrates RabbitMQ, Postgres, mitmproxy, and the app
- goreleaser + Dockerfile for CI/CD and releases

##### Component Interactions and Data Flow
1) User invokes CLI (binary or Docker container) with flags
2) CLI loads config → initializes logging → connects to DB and/or MQ (if requested)
3) Depending on input flags:
- Single-bucket or file input → scanner core
- -mq → RabbitMQ consumer (worker) streams bucket tasks to scanner core
4) Scanner core spins up N threads:
- For each bucket:
a. Provider client resolves endpoint & credentials
b. Issues ListBucketACL, ListBucket, GetBucketLocation calls
c. Optionally enumerates objects (ListObjects pages)
d. Aggregates permissions & object metadata
5) Results:
- Printed to console (human or JSON)
- Persisted to Postgres via GORM if -db is set
- (When mqingest subcommand is used, publishes JSON Bucket messages to RabbitMQ)

##### Architectural Patterns & Principles
- Modular packages for separation of concerns (bucket, permission, provider, db, mq)
- Dependency Injection: provider implementations selected at runtime via mapping
- Worker pool pattern for concurrent scanning threads
- Configuration-driven design (providers, MQ, DB settings)
- Auto-migration for data schema management

- A deployment view showing Docker Compose containers and external services

## Diagram (Mermaid)

```mermaid
graph TB

    %% CLI Frontend Layer
    subgraph "CLI Frontend"
        direction TB
        main["main.go"]:::frontend
        ss["s3scanner CLI"]:::frontend
        rc["regioncheck CLI"]:::frontend
        mqcli["mqingest CLI"]:::frontend
    end

    %% Scanner Core Layer
    subgraph "Scanner Core"
        direction TB
        core["Scanner Core"]:::core
        subgraph "Core Packages"
            direction TB
            bp["bucket package"]:::core
            pp["permission package"]:::core
            cp["collection package"]:::core
            gp["groups package"]:::core
        end
    end

    %% Provider Clients Layer
    subgraph "Provider Clients"
        direction TB
        cm["clientmap"]:::provider
        cred["credentials & endpoints"]:::provider
        aws["AWS Client"]:::provider
        gcp["GCP Client"]:::provider
        do["DigitalOcean Client"]:::provider
        dh["DreamHost Client"]:::provider
        ln["Linode Client"]:::provider
        sc["Scaleway Client"]:::provider
        wa["Wasabi Client"]:::provider
        cu["Custom Client"]:::provider
    end

    %% Messaging Layer
    subgraph "Messaging"
        direction TB
        mqPkg["mq package"]:::infra
        rabbit["RabbitMQ Broker"]:::infra
        worker["Worker"]:::infra
    end

    %% Persistence Layer
    subgraph "Persistence"
        direction TB
        dbPkg["db package"]:::db
        pg["PostgreSQL DB"]:::db
    end

    %% Utilities Layer
    subgraph "Utilities"
        direction TB
        config["config.yml"]:::util
        logpkg["Logging package"]:::util
    end

    %% Dev & Packaging Layer
    subgraph "Dev & Packaging"
        direction TB
        compose[".dev/docker-compose.yml"]:::packaging
        dockerfile["Dockerfile"]:::packaging
        goreleaser[".goreleaser.yaml"]:::packaging
    end

    %% Connections
    main -->|invokes| ss
    main -->|invokes| rc
    main -->|invokes| mqcli
    main -->|loads config| config
    main -->|initializes logging| logpkg

    ss -->|starts scan| core
    rc -->|validates regions| core
    mqcli -->|publishes buckets| mqPkg
    mqPkg -->|pushes to| rabbit
    rabbit -->|consumes| worker
    worker -->|feeds bucket to| core

    core -->|uses| bp
    core -->|uses| pp
    core -->|uses| cp
    core -->|uses| gp

    core -->|resolves client| cm
    cm --> aws
    cm --> gcp
    cm --> do
    cm --> dh
    cm --> ln
    cm --> sc
    cm --> wa
    cm --> cu
    core -->|loads credentials| cred

    core -->|persists results| dbPkg
    dbPkg -->|writes to| pg

    core -->|loads config| config
    core -->|logs events| logpkg

    compose --> rabbit
    compose --> pg
    compose -->|runs app container| main
    compose -->|runs app container| worker
    compose -->|runs app container| core

    dockerfile --> compose
    goreleaser --> dockerfile

    %% Click Events
    click main "https://github.com/sa7mon/s3scanner/blob/main/main.go"
    click ss "https://github.com/sa7mon/s3scanner/blob/main/cmd/s3scanner/s3scanner.go"
    click rc "https://github.com/sa7mon/s3scanner/blob/main/cmd/regioncheck/regioncheck.go"
    click mqcli "https://github.com/sa7mon/s3scanner/blob/main/cmd/mqingest/mqingest.go"
    click bp "https://github.com/sa7mon/s3scanner/blob/main/bucket/bucket.go"
    click pp "https://github.com/sa7mon/s3scanner/blob/main/permission/permission.go"
    click cp "https://github.com/sa7mon/s3scanner/blob/main/collection/collection.go"
    click gp "https://github.com/sa7mon/s3scanner/blob/main/groups/groups.go"
    click aws "https://github.com/sa7mon/s3scanner/blob/main/provider/aws.go"
    click gcp "https://github.com/sa7mon/s3scanner/blob/main/provider/gcp.go"
    click do "https://github.com/sa7mon/s3scanner/blob/main/provider/digitalocean.go"
    click dh "https://github.com/sa7mon/s3scanner/blob/main/provider/dreamhost.go"
    click ln "https://github.com/sa7mon/s3scanner/blob/main/provider/linode.go"
    click sc "https://github.com/sa7mon/s3scanner/blob/main/provider/scaleway.go"
    click wa "https://github.com/sa7mon/s3scanner/blob/main/provider/wasabi.go"
    click cu "https://github.com/sa7mon/s3scanner/blob/main/provider/custom.go"
    click cm "https://github.com/sa7mon/s3scanner/blob/main/provider/clientmap/clientmap.go"
    click cred "https://github.com/sa7mon/s3scanner/blob/main/provider/credentials.go"
    click dbPkg "https://github.com/sa7mon/s3scanner/blob/main/db/database.go"
    click mqPkg "https://github.com/sa7mon/s3scanner/blob/main/mq/mq.go"
    click worker "https://github.com/sa7mon/s3scanner/blob/main/worker/mq_worker.go"
    click config "https://github.com/sa7mon/s3scanner/blob/main/config.yml"
    click logpkg "https://github.com/sa7mon/s3scanner/blob/main/log/formatter_hook.go"
    click logpkg "https://github.com/sa7mon/s3scanner/blob/main/log/nested_json_formatter.go"
    click compose "https://github.com/sa7mon/s3scanner/blob/main/.dev/docker-compose.yml"
    click dockerfile "https://github.com/sa7mon/s3scanner/tree/main/packaging/docker/Dockerfile"
    click goreleaser "https://github.com/sa7mon/s3scanner/blob/main/packaging/.goreleaser.yaml"

    %% Styles
    classDef frontend fill:#D6E9FF,stroke:#0366D6;
    classDef core fill:#CFE2F3,stroke:#0B5394;
    classDef provider fill:#D5E8D4,stroke:#82B366;
    classDef infra fill:#FFE6CC,stroke:#BF9000;
    classDef db fill:#E1D5E7,stroke:#9673A6;
    classDef util fill:#F2F2F2,stroke:#666666;
    classDef packaging fill:#FDE2E2,stroke:#A41E11;
```
