# drivendataorg/cloudpathlib architecture

> Architecture diagram of the GitHub repository drivendataorg/cloudpathlib, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/drivendataorg/cloudpathlib
- Repository: https://github.com/drivendataorg/cloudpathlib

## Overview

##### Project type and purpose
- Type: A Python library / open-source tool (not a full-stack app).
- Purpose: Expose a uniform `pathlib.Path`-style API (`CloudPath`) for interacting with AWS S3, Google Cloud Storage, Azure Blob Storage, and HTTP URIs, with seamless local caching and testability.

##### Key structural elements from the file tree
- Core package: `cloudpathlib/`
– Base modules: `anypath.py`, `cloudpath.py`, `client.py`, `enums.py`, `exceptions.py`, `url_utils.py`
– Subpackages by scheme:
· `s3/` → `S3Path`, `S3Client`
· `gs/` → `GSPath`, `GSClient`
· `azure/` → `AzureBlobPath`, `AzureBlobClient`
· `http/` → `HttpPath`, `HttpClient`
· `local/` → local‐FS fallback & cache management
· `legacy/` → backwards‐compatibility helpers
- Tests: comprehensive suite under `tests/` (not in production flow)
- Documentation: `docs/` driven by MkDocs

##### Main components
A. Public API Layer
– `CloudPath` factory / entrypoint
B. Parser & Dispatch
– Parses URI scheme (`s3://`, `gs://`, `azure://`, `http://`), returns a scheme-specific Path class
C. Path Classes (one per supported scheme)
– Implement `pathlib` methods (`open`, `read_text`, `glob`, etc.)
– Delegate cloud operations to a corresponding Client
D. Client Classes
– Wrap the cloud SDK or HTTP library calls (`boto3`, `google-cloud-storage`, `azure-storage-blob`, `requests`)
– Handle low-level upload/download, metadata fetching, pagination, authentication
E. Caching / Local FileSystem Layer
– Local cache directory on disk (configurable)
– Ensures files are downloaded only when needed, reused across calls/processes
F. External Services
– AWS S3 service
– Google Cloud Storage service
– Azure Blob Storage service
– Generic HTTP servers (for `https://` URIs)

##### Relationships & data flows
1. User code calls `CloudPath("s3://...")` → CloudPath factory
2. Factory *parses* the URI → returns `S3Path` instance
3. Calling `path.read_text()` or `path.open("r")`
a. `S3Path` requests its `S3Client` to download the object to cache
b. Client uses `boto3` under the hood
c. The file lands in a local cache folder → local file I/O returns content
4. Writing (`write_text`, `upload_from`) reverses: local temp file → `S3Client.upload_from()` → calls `upload_file` in `boto3` → object in cloud
5. `glob()`, `iterdir()`, `exists()`, etc., map to corresponding client list/stat API calls

##### Important architectural patterns & design principles
- Factory / Dispatch pattern (`CloudPath.from_uri`) to pick the right Path class
- Strategy pattern: each `Path` class delegates to a corresponding `Client` class
- Adapter / Facade: wrap each cloud SDK in a uniform interface
- Separation of concerns: Path vs. Client vs. caching vs. parsing
- Extensibility: add a new service by supplying two small classes (`MyPath`, `MyClient`)

– Optionally annotate key methods at each boundary (e.g., `CloudPath.from_uri()`, `path.open()`, `client.download()`)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Public API & Factory
    subgraph "Public API & Factory"
        CPF["CloudPath Factory"]:::core
        UPD["URI Parser & Dispatch"]:::core
    end

    %% Core Utilities
    subgraph "Core Utilities"
        BC["Base Client Abstractions"]:::core
        UU["URL Utilities"]:::core
        EE["Enums & Exceptions"]:::core
    end

    %% Service Modules
    subgraph "Service Modules"
        S3P["S3Path"]:::service
        S3C["S3Client"]:::service
        GSP["GSPath"]:::service
        GSC["GSClient"]:::service
        AZP["AzureBlobPath"]:::service
        AZC["AzureBlobClient"]:::service
        HTP["HttpPath"]:::service
        HTC["HttpClient"]:::service
    end

    %% Local Filesystem & Caching
    subgraph "Local Filesystem & Caching"
        LP["LocalPath"]:::cache
        LC["LocalClient"]:::cache
        subgraph "Service-specific Local Implementations"
            LSI1["S3 Implementation"]:::cache
            LSI2["GS Implementation"]:::cache
            LSI3["Azure Implementation"]:::cache
        end
    end

    %% Legacy Helpers
    LH["Legacy Helpers"]:::core

    %% External Services
    subgraph "External Services"
        AWS["AWS S3 Service"]:::external
        GCS["Google Cloud Storage Service"]:::external
        AZB["Azure Blob Storage Service"]:::external
        HTTP["HTTP Servers"]:::external
    end

    %% Connections
    CPF --> UPD
    UPD --> S3P
    UPD --> GSP
    UPD --> AZP
    UPD --> HTP

    S3P --> S3C
    GSP --> GSC
    AZP --> AZC
    HTP --> HTC

    S3C --> LP
    GSC --> LP
    AZC --> LP
    HTC --> LP

    S3C --> AWS
    GSC --> GCS
    AZC --> AZB
    HTC --> HTTP

    LP --> LC
    LC --> LSI1
    LC --> LSI2
    LC --> LSI3

    CPF --> BC
    CPF --> UU
    CPF --> EE
    LH --> CPF

    %% Click Events
    click CPF "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/cloudpath.py"
    click UPD "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/anypath.py"
    click BC "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/client.py"
    click UU "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/url_utils.py"
    click EE "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/enums.py"
    click EE "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/exceptions.py"
    click S3P "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/s3/s3path.py"
    click S3C "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/s3/s3client.py"
    click GSP "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/gs/gspath.py"
    click GSC "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/gs/gsclient.py"
    click AZP "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/azure/azblobpath.py"
    click AZC "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/azure/azblobclient.py"
    click HTP "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/http/httppath.py"
    click HTC "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/http/httpclient.py"
    click LP "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/local/localpath.py"
    click LC "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/local/localclient.py"
    click LSI1 "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/local/implementations/s3.py"
    click LSI2 "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/local/implementations/gs.py"
    click LSI3 "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/local/implementations/azure.py"
    click LH "https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/legacy/glob.py"

    %% Styles
    classDef core fill:#AED6F1,stroke:#2980B9,color:#154360
    classDef service fill:#ABEBC6,stroke:#27AE60,color:#145A32
    classDef cache fill:#FCF3CF,stroke:#F1C40F,color:#7D6608
    classDef external fill:#F5B7B1,stroke:#E74C3C,color:#641E16
```
