How fyoorer/shadowclone works
ShadowClone is a CLI for splitting line-oriented input, distributing command executions across serverless functions, and collecting their output. Its documented workflow supports an additional unsplit file and cloud-provider-agnostic execution through Lithops. The excerpts show the CLI, chunk handling, object-storage operations, and worker commands in one source file; unsampled code is not expanded into guessed subsystems.
Main entry points: shadowclone.py CLI orchestration · hasher.py Chunk hashing · config.py Runtime config
Components 10
CLI workflow
- CLI orchestration
shadowclone.py - Input splitting
shadowclone.py - Result output
shadowclone.py
Distributed execution
- Cloud workers
shadowclone.py - User commands
- Lithops executor
Configuration and storage
- Runtime config
config.py - Object storage
- Chunk hashing
hasher.py
Other components
- CLI user
Connections 11
- CLI user → CLI orchestration: runs
README.md - CLI orchestration → Runtime config: reads settings
shadowclone.py - CLI orchestration → Input splitting: splits input
shadowclone.py - CLI orchestration → Object storage: uploads chunks
shadowclone.py - CLI orchestration → Lithops executor: maps workers
shadowclone.py - Lithops executor → Cloud workers: dispatches
shadowclone.py - Cloud workers → Object storage: downloads input
shadowclone.py - Cloud workers → User commands: runs
shadowclone.py - CLI orchestration → Result output: writes results
shadowclone.py - CLI orchestration → Chunk hashing: hashes chunks
shadowclone.py - Chunk hashing → Object storage: deduplicates uploads