How sa7mon/s3scanner works
Project Type and Purpose
• S3Scanner is a command-line, multi-threaded scanning tool (written in Go) for discovering open S3 buckets across AWS and other cloud providers. It can ingest bucket names via CLI, file, or RabbitMQ, check permissions and optionally enumerate objects, then output results to stdout or persist them in a PostgreSQL database.
High-Level Architecture and Major Components
a. CLI Frontend
• Entry point (main.go + cmd/…) handling subcommands:
– s3scanner (bucket scan)
– regioncheck (validate provider regions)
– mqingest (publish buckets into RabbitMQ)
• Parses flags (-bucket, -mq, -db, -provider, -threads, etc.) and loads config.yml.
b. Scanner Core
• bucket package: represents bucket objects and metadata
• permission package: checks ACLs (read/write/ACP/full control)
• collection & groups: processes per-bucket results and groups findings
• enumeration logic (optional, single-threaded per bucket)
c. Provider Clients
• provider package: concrete implementations for AWS, GCP, DigitalOcean, Linode, DreamHost, Scaleway, Wasabi, and “custom.”
• clientmap: dynamic mapping of provider names to client constructors
• credentials & endpoint resolution via config.yml
d. Messaging Layer (RabbitMQ)
• mq package: connects to RabbitMQ using mq.uri and mq.queue_name
• Worker component (worker/mq_worker.go) consumes bucket names and pushes them into the Scanner Core
e. Persistence Layer (PostgreSQL via GORM)
• db package: opens a GORM connection, auto-migrates schema, and writes scan results
• Entities: bucket existence, region, permission details, enumeration results
f. Supporting Utilities
• log package: hooks and JSON formatter for structured logging
• config management: config.yml file resolution (current dir, /etc/s3scanner, $HOME/.s3scanner)
g. Dev & Packaging
• Docker Compose (.dev/docker-compose.yml) orchestrates RabbitMQ, Postgres, mitmproxy, and the app
• goreleaser + Dockerfile for CI/CD and releases
Component Interactions and Data Flow
1) User invokes CLI (binary or Docker container) with flags
2) CLI loads config → initializes logging → connects to DB and/or MQ (if requested)
3) Depending on input flags:
• Single-bucket or file input → scanner core
• -mq → RabbitMQ consumer (worker) streams bucket tasks to scanner core
4) Scanner core spins up N threads:
• For each bucket:
a. Provider client resolves endpoint & credentials
b. Issues ListBucketACL, ListBucket, GetBucketLocation calls
c. Optionally enumerates objects (ListObjects pages)
d. Aggregates permissions & object metadata
5) Results:
• Printed to console (human or JSON)
• Persisted to Postgres via GORM if -db is set
• (When mqingest subcommand is used, publishes JSON Bucket messages to RabbitMQ)
Architectural Patterns & Principles
• Modular packages for separation of concerns (bucket, permission, provider, db, mq)
• Dependency Injection: provider implementations selected at runtime via mapping
• Worker pool pattern for concurrent scanning threads
• Configuration-driven design (providers, MQ, DB settings)
• Auto-migration for data schema management
• A deployment view showing Docker Compose containers and external services