How drivendataorg/cloudpathlib works
Project type and purpose
• Type: A Python library / open-source tool (not a full-stack app).
• Purpose: Expose a uniform pathlib.Path-style API (CloudPath) for interacting with AWS S3, Google Cloud Storage, Azure Blob Storage, and HTTP URIs, with seamless local caching and testability.
Key structural elements from the file tree
• Core package: cloudpathlib/
– Base modules: anypath.py, cloudpath.py, client.py, enums.py, exceptions.py, url_utils.py
– Subpackages by scheme:
· s3/ → S3Path, S3Client
· gs/ → GSPath, GSClient
· azure/ → AzureBlobPath, AzureBlobClient
· http/ → HttpPath, HttpClient
· local/ → local‐FS fallback & cache management
· legacy/ → backwards‐compatibility helpers
• Tests: comprehensive suite under tests/ (not in production flow)
• Documentation: docs/ driven by MkDocs
Main components
A. Public API Layer
– CloudPath factory / entrypoint
B. Parser & Dispatch
– Parses URI scheme (s3://, gs://, azure://, http://), returns a scheme-specific Path class
C. Path Classes (one per supported scheme)
– Implement pathlib methods (open, read_text, glob, etc.)
– Delegate cloud operations to a corresponding Client
D. Client Classes
– Wrap the cloud SDK or HTTP library calls (boto3, google-cloud-storage, azure-storage-blob, requests)
– Handle low-level upload/download, metadata fetching, pagination, authentication
E. Caching / Local FileSystem Layer
– Local cache directory on disk (configurable)
– Ensures files are downloaded only when needed, reused across calls/processes
F. External Services
– AWS S3 service
– Google Cloud Storage service
– Azure Blob Storage service
– Generic HTTP servers (for https:// URIs)
Relationships & data flows
1. User code calls CloudPath("s3://...") → CloudPath factory
2. Factory *parses* the URI → returns S3Path instance
3. Calling path.read_text() or path.open("r")
a. S3Path requests its S3Client to download the object to cache
b. Client uses boto3 under the hood
c. The file lands in a local cache folder → local file I/O returns content
4. Writing (write_text, upload_from) reverses: local temp file → S3Client.upload_from() → calls upload_file in boto3 → object in cloud
5. glob(), iterdir(), exists(), etc., map to corresponding client list/stat API calls
Important architectural patterns & design principles
• Factory / Dispatch pattern (CloudPath.from_uri) to pick the right Path class
• Strategy pattern: each Path class delegates to a corresponding Client class
• Adapter / Facade: wrap each cloud SDK in a uniform interface
• Separation of concerns: Path vs. Client vs. caching vs. parsing
• Extensibility: add a new service by supplying two small classes (MyPath, MyClient)
– Optionally annotate key methods at each boundary (e.g., CloudPath.from_uri(), path.open(), client.download())