# lfnovo/omni-storage architecture

> Architecture diagram of the GitHub repository lfnovo/omni-storage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lfnovo/omni-storage
- Repository: https://github.com/lfnovo/omni-storage

## Overview

1. **What this repo is**
   - `omni_storage` is a small Python library that hides the differences between three storage backends: local filesystem, Amazon S3, and Google Cloud Storage.
   - The architecture is centered on a single abstract storage API, with concrete adapters per backend.

2. **Top-level package boundary**
   - All runtime code lives under `src/omni_storage`.
   - The package is intentionally thin: it exposes a common interface and delegates provider-specific behavior to backend classes.

3. **Core abstraction**
   - `base.py` defines the `Storage` abstract base class and the shared contract for file operations.
   - The supported operations are the architectural backbone: save, read, upload, existence checks, URL generation, and append.
   - This creates the stable boundary that callers depend on, regardless of backend.

4. **Backend adapters**
   - `local.py` implements `Storage` against the local filesystem and likely uses direct file I/O for the simplest path.
   - `s3.py` and `gcs.py` implement the same contract using cloud SDKs and bucket/object semantics.
   - These are the main substitution points: the rest of the code should not need to know which backend is active.

5. **Factory and backend selection**
   - `factory.py` is the runtime selection layer.
   - It chooses the backend either explicitly via `storage_type` or implicitly from environment variables such as `AWS_S3_BUCKET`, `GCS_BUCKET`, and `DATADIR`.
   - This means configuration is externalized; the app using the library can switch storage without changing call sites.

6. **Append architecture**
   - `manifest.py` indicates a special append subsystem rather than a naive overwrite model.
   - The README describes two append strategies for cloud storage: a simple single-object strategy for smaller data and a multipart/manifest strategy for larger data.
   - This is a key architectural feature because append is not uniformly supported across backends, so the library normalizes it with an internal coordination layer.

7. **Data flow through the system**
   - A caller obtains a `Storage` instance from the factory.
   - File data then flows into backend methods as bytes or file-like objects, and the backend returns paths/URLs or append results.
   - For cloud append operations, the flow may expand into part objects plus a manifest that later reconstructs the logical file on read.

8. **Shared types and typing discipline**
   - `types.py` centralizes cross-cutting type definitions, including append-related result structures and literal strategy values.
   - The presence of `py.typed` means the package is published as typed and intended to be consumed with static analysis support.

9. **Repository layout and modularity**
   - The repo follows a clean “interface + adapters + factory” structure.
   - There is no sign of a web framework, worker queue, or database layer; the primary integration surface is file storage backends and environment configuration.

10. **Testing architecture**
   - `tests/` mirrors the runtime modules: base behavior, factory selection, manifest logic, local backend, S3 backend, GCS backend, and append behavior.
   - This implies the test strategy is contract-driven: common behavior is validated once, while each adapter is tested against its backend-specific semantics.

11. **Runtime and dependency model**
   - The project uses modern Python packaging with `pyproject.toml` and `uv.lock`.
   - Optional extras isolate cloud dependencies, so the local backend can be used without installing S3/GCS SDKs.
   - That separation matters architecturally because backend availability is a dependency concern, not a code-path concern.

12. **Operational and delivery boundary**
   - GitHub Actions handles test, publish, and code-review automation.
   - From an architecture-diagram perspective, these are external lifecycle concerns rather than runtime components, but they reinforce that this is a packaged library, not a deployed service.

13. **How to draw the diagram**
   - Put `Storage` in the center as the abstract contract.
   - Show three concrete adapters behind it: Local, S3, and GCS.
   - Place `factory.py` above them as the configuration-based selector.
   - Add `manifest.py` as an internal helper used mainly by append/read flows for cloud backends.
   - Surround the whole library with environment variables and cloud SDK/bucket services as the external dependencies.

Main entry points: [`src/omni_storage`](https://github.com/lfnovo/omni-storage/tree/main/src/omni_storage) (omni_storage), [`src/omni_storage/base.py`](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/base.py) (Storage), [`src/omni_storage/s3.py`](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/s3.py) (S3 storage), [`src/omni_storage/gcs.py`](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/gcs.py) (GCS storage)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime package"]
  node_package["omni_storage"]
  node_init["API export<br/>[__init__.py]"]
  node_base["Storage<br/>abstract api<br/>[base.py]"]
  node_factory["Factory<br/>selector<br/>[factory.py]"]
  node_local["Local storage<br/>adapter<br/>[local.py]"]
  node_s3["S3 storage<br/>adapter<br/>[s3.py]"]
  node_gcs["GCS storage<br/>adapter<br/>[gcs.py]"]
  node_manifest["Manifest<br/>append coordinator<br/>[manifest.py]"]
  node_types["Types<br/>typing<br/>[types.py]"]
end

subgraph group_external["External systems"]
  node_env(("Env config<br/>configuration"))
  node_fs[("Filesystem<br/>local backend")]
  node_aws[("AWS S3<br/>cloud service")]
  node_gcp[("GCS<br/>cloud service")]
  node_aws_sdk{{"AWS SDK<br/>dependency"}}
  node_gcs_sdk{{"GCS SDK<br/>dependency"}}
  node_caller(("Library users<br/>consumer"))
end

subgraph group_tests["Tests"]
  node_test_base["Base tests<br/>test suite<br/>[test_base.py]"]
  node_test_factory["Factory tests<br/>test suite<br/>[test_factory.py]"]
  node_test_local["Local tests<br/>test suite"]
  node_test_s3["S3 tests<br/>test suite<br/>[test_s3_storage.py]"]
  node_test_gcs["GCS tests<br/>test suite"]
  node_test_manifest["Manifest tests<br/>test suite<br/>[test_manifest.py]"]
  node_test_append["Append tests<br/>test suite<br/>[test_append.py]"]
end

node_package -->|"exports"| node_init
node_package -->|"defines"| node_base
node_package -->|"selects"| node_factory
node_package -->|"implements"| node_local
node_package -->|"implements"| node_s3
node_package -->|"implements"| node_gcs
node_package -->|"coordinates"| node_manifest
node_package -->|"shares"| node_types
node_factory -->|"reads"| node_env
node_factory -->|"returns"| node_base
node_local -->|"conforms to"| node_base
node_s3 -->|"conforms to"| node_base
node_gcs -->|"conforms to"| node_base
node_local -->|"uses"| node_fs
node_s3 -->|"uses"| node_aws_sdk
node_s3 -->|"stores in"| node_aws
node_gcs -->|"uses"| node_gcs_sdk
node_gcs -->|"stores in"| node_gcp
node_manifest -->|"returns"| node_types
node_s3 -->|"append flow"| node_manifest
node_gcs -->|"append flow"| node_manifest
node_caller -->|"obtains"| node_factory
node_caller -->|"depends on"| node_base
node_test_base -->|"verifies"| node_base
node_test_factory -->|"verifies"| node_factory
node_test_local -->|"verifies"| node_local
node_test_s3 -->|"verifies"| node_s3
node_test_gcs -->|"verifies"| node_gcs
node_test_manifest -->|"verifies"| node_manifest
node_test_append -->|"covers"| node_manifest
node_test_append -->|"covers"| node_base

click node_package "https://github.com/lfnovo/omni-storage/tree/main/src/omni_storage"
click node_init "https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/__init__.py"
click node_base "https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/base.py"
click node_factory "https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/factory.py"
click node_local "https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/local.py"
click node_s3 "https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/s3.py"
click node_gcs "https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/gcs.py"
click node_manifest "https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/manifest.py"
click node_types "https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/types.py"
click node_test_base "https://github.com/lfnovo/omni-storage/blob/main/tests/test_base.py"
click node_test_factory "https://github.com/lfnovo/omni-storage/blob/main/tests/test_factory.py"
click node_test_local "https://github.com/lfnovo/omni-storage/blob/main/tests/test_local_storage.py"
click node_test_s3 "https://github.com/lfnovo/omni-storage/blob/main/tests/test_s3_storage.py"
click node_test_gcs "https://github.com/lfnovo/omni-storage/blob/main/tests/test_gcs_storage.py"
click node_test_manifest "https://github.com/lfnovo/omni-storage/blob/main/tests/test_manifest.py"
click node_test_append "https://github.com/lfnovo/omni-storage/blob/main/tests/test_append.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_package,node_init,node_base,node_factory,node_local,node_s3,node_gcs,node_manifest,node_types toneBlue
class node_env,node_fs,node_aws,node_gcp,node_aws_sdk,node_gcs_sdk,node_caller toneAmber
class node_test_base,node_test_factory,node_test_local,node_test_s3,node_test_gcs,node_test_manifest,node_test_append toneMint
```

## Components

### Runtime package

Core library code under src/omni_storage.

- [omni_storage](https://github.com/lfnovo/omni-storage/tree/main/src/omni_storage): package · Thin storage abstraction library. · `src/omni_storage`
- [API export](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/__init__.py): module · Package entrypoint exposing the public interface. · `src/omni_storage/__init__.py`
- [Storage](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/base.py): abstract api · Shared storage contract for save, read, upload, exists, URL, and append. · `src/omni_storage/base.py`
- [Factory](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/factory.py): selector · Chooses a storage backend from config or environment. · `src/omni_storage/factory.py`
- [Local storage](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/local.py): adapter · Filesystem-backed Storage implementation. · `src/omni_storage/local.py`
- [S3 storage](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/s3.py): adapter · Amazon S3-backed Storage implementation. · `src/omni_storage/s3.py`
- [GCS storage](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/gcs.py): adapter · Google Cloud Storage-backed Storage implementation. · `src/omni_storage/gcs.py`
- [Manifest](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/manifest.py): append coordinator · Append coordination for multipart or reconstructed cloud writes. · `src/omni_storage/manifest.py`
- [Types](https://github.com/lfnovo/omni-storage/blob/main/src/omni_storage/types.py): typing · Shared type aliases and append result structures. · `src/omni_storage/types.py`

### External systems

Environment and backend services used by storage adapters.

- **Env config**: configuration · Storage selection inputs such as AWS_S3_BUCKET, GCS_BUCKET, and DATADIR.
- **Filesystem**: local backend · Local file system used by the local adapter.
- **AWS S3**: cloud service · Object storage service used by the S3 adapter.
- **GCS**: cloud service · Object storage service used by the GCS adapter.
- **AWS SDK**: dependency · Python client dependency for S3 operations.
- **GCS SDK**: dependency · Python client dependency for GCS operations.
- **Library users**: consumer · Applications that depend on the common Storage API.

### Tests

Contract and backend-specific test coverage.

- [Base tests](https://github.com/lfnovo/omni-storage/blob/main/tests/test_base.py): test suite · Contract coverage for the shared Storage behavior. · `tests/test_base.py`
- [Factory tests](https://github.com/lfnovo/omni-storage/blob/main/tests/test_factory.py): test suite · Coverage for backend selection and config handling. · `tests/test_factory.py`
- [Local tests](https://github.com/lfnovo/omni-storage/blob/main/tests/test_local_storage.py): test suite · Behavior tests for the filesystem adapter. · `tests/test_local_storage.py`
- [S3 tests](https://github.com/lfnovo/omni-storage/blob/main/tests/test_s3_storage.py): test suite · Behavior tests for the S3 adapter. · `tests/test_s3_storage.py`
- [GCS tests](https://github.com/lfnovo/omni-storage/blob/main/tests/test_gcs_storage.py): test suite · Behavior tests for the GCS adapter. · `tests/test_gcs_storage.py`
- [Manifest tests](https://github.com/lfnovo/omni-storage/blob/main/tests/test_manifest.py): test suite · Coverage for append coordination and manifest behavior. · `tests/test_manifest.py`
- [Append tests](https://github.com/lfnovo/omni-storage/blob/main/tests/test_append.py): test suite · End-to-end append behavior across supported backends. · `tests/test_append.py`

## Connections

- omni_storage → API export: exports · Public package surface re-exports the storage API.
- omni_storage → Storage: defines · Contains the abstract Storage contract.
- omni_storage → Factory: selects · Factory constructs backend implementations.
- omni_storage → Local storage: implements · Local filesystem adapter.
- omni_storage → S3 storage: implements · S3 adapter.
- omni_storage → GCS storage: implements · GCS adapter.
- omni_storage → Manifest: coordinates · Append coordination helper.
- omni_storage → Types: shares · Common type definitions used across modules.
- Factory → Env config: reads · Backend selection comes from environment and config.
- Factory → Storage: returns · Builds a Storage implementation.
- Local storage → Storage: conforms to · Implements the shared Storage API.
- S3 storage → Storage: conforms to · Implements the shared Storage API.
- GCS storage → Storage: conforms to · Implements the shared Storage API.
- Local storage → Filesystem: uses · Persists data on the local filesystem.
- S3 storage → AWS SDK: uses · Depends on the S3 client library.
- S3 storage → AWS S3: stores in · Targets S3 buckets and objects.
- GCS storage → GCS SDK: uses · Depends on the GCS client library.
- GCS storage → GCS: stores in · Targets GCS buckets and objects.
- Manifest → Types: returns · Uses append result and strategy types.
- S3 storage → Manifest: append flow · Cloud append can route through manifest coordination.
- GCS storage → Manifest: append flow · Cloud append can route through manifest coordination.
- Library users → Factory: obtains · Users request a backend through the factory.
- Library users → Storage: depends on · Callers program against the abstract Storage interface.
- Base tests → Storage: verifies · Checks shared contract behavior.
- Factory tests → Factory: verifies · Checks backend selection logic.
- Local tests → Local storage: verifies · Checks filesystem backend behavior.
- S3 tests → S3 storage: verifies · Checks S3 backend behavior.
- GCS tests → GCS storage: verifies · Checks GCS backend behavior.
- Manifest tests → Manifest: verifies · Checks append coordination behavior.
- Append tests → Manifest: covers · Exercises append behavior across backends.
- Append tests → Storage: covers · Validates append through the shared API.
