How lfnovo/omni-storage works
1. What this repo is
- omni_storage is a small Python library that hides the differences between three storage backends: local filesystem, Amazon S3, and Google Cloud Storage.
- The architecture is centered on a single abstract storage API, with concrete adapters per backend.
2. Top-level package boundary
- All runtime code lives under src/omni_storage.
- The package is intentionally thin: it exposes a common interface and delegates provider-specific behavior to backend classes.
3. Core abstraction
- base.py defines the Storage abstract base class and the shared contract for file operations.
- The supported operations are the architectural backbone: save, read, upload, existence checks, URL generation, and append.
- This creates the stable boundary that callers depend on, regardless of backend.
4. Backend adapters
- local.py implements Storage against the local filesystem and likely uses direct file I/O for the simplest path.
- s3.py and gcs.py implement the same contract using cloud SDKs and bucket/object semantics.
- These are the main substitution points: the rest of the code should not need to know which backend is active.
5. Factory and backend selection
- factory.py is the runtime selection layer.
- It chooses the backend either explicitly via storage_type or implicitly from environment variables such as AWS_S3_BUCKET, GCS_BUCKET, and DATADIR.
- This means configuration is externalized; the app using the library can switch storage without changing call sites.
6. Append architecture
- manifest.py indicates a special append subsystem rather than a naive overwrite model.
- The README describes two append strategies for cloud storage: a simple single-object strategy for smaller data and a multipart/manifest strategy for larger data.
- This is a key architectural feature because append is not uniformly supported across backends, so the library normalizes it with an internal coordination layer.
7. Data flow through the system
- A caller obtains a Storage instance from the factory.
- File data then flows into backend methods as bytes or file-like objects, and the backend returns paths/URLs or append results.
- For cloud append operations, the flow may expand into part objects plus a manifest that later reconstructs the logical file on read.
8. Shared types and typing discipline
- types.py centralizes cross-cutting type definitions, including append-related result structures and literal strategy values.
- The presence of py.typed means the package is published as typed and intended to be consumed with static analysis support.
9. Repository layout and modularity
- The repo follows a clean “interface + adapters + factory” structure.
- There is no sign of a web framework, worker queue, or database layer; the primary integration surface is file storage backends and environment configuration.
10. Testing architecture
- tests/ mirrors the runtime modules: base behavior, factory selection, manifest logic, local backend, S3 backend, GCS backend, and append behavior.
- This implies the test strategy is contract-driven: common behavior is validated once, while each adapter is tested against its backend-specific semantics.
11. Runtime and dependency model
- The project uses modern Python packaging with pyproject.toml and uv.lock.
- Optional extras isolate cloud dependencies, so the local backend can be used without installing S3/GCS SDKs.
- That separation matters architecturally because backend availability is a dependency concern, not a code-path concern.
12. Operational and delivery boundary
- GitHub Actions handles test, publish, and code-review automation.
- From an architecture-diagram perspective, these are external lifecycle concerns rather than runtime components, but they reinforce that this is a packaged library, not a deployed service.
13. How to draw the diagram
- Put Storage in the center as the abstract contract.
- Show three concrete adapters behind it: Local, S3, and GCS.
- Place factory.py above them as the configuration-based selector.
- Add manifest.py as an internal helper used mainly by append/read flows for cloud backends.
- Surround the whole library with environment variables and cloud SDK/bucket services as the external dependencies.
Main entry points: omni_storage omni_storage · base.py Storage · s3.py S3 storage · gcs.py GCS storage
Components 23
Runtime package
Core library code under src/omni_storage.
- omni_storage · Thin storage abstraction library.
src/omni_storage - API export · Package entrypoint exposing the public interface.
src/omni_storage/__init__.py - Storage · Shared storage contract for save, read, upload, exists, URL, and append.
src/omni_storage/base.py - Factory · Chooses a storage backend from config or environment.
src/omni_storage/factory.py - Local storage · Filesystem-backed Storage implementation.
src/omni_storage/local.py - S3 storage · Amazon S3-backed Storage implementation.
src/omni_storage/s3.py - GCS storage · Google Cloud Storage-backed Storage implementation.
src/omni_storage/gcs.py - Manifest · Append coordination for multipart or reconstructed cloud writes.
src/omni_storage/manifest.py - Types · Shared type aliases and append result structures.
src/omni_storage/types.py
External systems
Environment and backend services used by storage adapters.
- Env config · Storage selection inputs such as AWS_S3_BUCKET, GCS_BUCKET, and DATADIR.
- Filesystem · Local file system used by the local adapter.
- AWS S3 · Object storage service used by the S3 adapter.
- GCS · Object storage service used by the GCS adapter.
- AWS SDK · Python client dependency for S3 operations.
- GCS SDK · Python client dependency for GCS operations.
- Library users · Applications that depend on the common Storage API.
Tests
Contract and backend-specific test coverage.
- Base tests · Contract coverage for the shared Storage behavior.
tests/test_base.py - Factory tests · Coverage for backend selection and config handling.
tests/test_factory.py - Local tests · Behavior tests for the filesystem adapter.
tests/test_local_storage.py - S3 tests · Behavior tests for the S3 adapter.
tests/test_s3_storage.py - GCS tests · Behavior tests for the GCS adapter.
tests/test_gcs_storage.py - Manifest tests · Coverage for append coordination and manifest behavior.
tests/test_manifest.py - Append tests · End-to-end append behavior across supported backends.
tests/test_append.py
Connections 31
- omni_storage → API export: exports · Public package surface re-exports the storage API.
- omni_storage → Storage: defines · Contains the abstract Storage contract.
- omni_storage → Factory: selects · Factory constructs backend implementations.
- omni_storage → Local storage: implements · Local filesystem adapter.
- omni_storage → S3 storage: implements · S3 adapter.
- omni_storage → GCS storage: implements · GCS adapter.
- omni_storage → Manifest: coordinates · Append coordination helper.
- omni_storage → Types: shares · Common type definitions used across modules.
- Factory → Env config: reads · Backend selection comes from environment and config.
- Factory → Storage: returns · Builds a Storage implementation.
- Local storage → Storage: conforms to · Implements the shared Storage API.
- S3 storage → Storage: conforms to · Implements the shared Storage API.
- GCS storage → Storage: conforms to · Implements the shared Storage API.
- Local storage → Filesystem: uses · Persists data on the local filesystem.
- S3 storage → AWS SDK: uses · Depends on the S3 client library.
- S3 storage → AWS S3: stores in · Targets S3 buckets and objects.
- GCS storage → GCS SDK: uses · Depends on the GCS client library.
- GCS storage → GCS: stores in · Targets GCS buckets and objects.
- Manifest → Types: returns · Uses append result and strategy types.
- S3 storage → Manifest: append flow · Cloud append can route through manifest coordination.
- GCS storage → Manifest: append flow · Cloud append can route through manifest coordination.
- Library users → Factory: obtains · Users request a backend through the factory.
- Library users → Storage: depends on · Callers program against the abstract Storage interface.
- Base tests → Storage: verifies · Checks shared contract behavior.
- Factory tests → Factory: verifies · Checks backend selection logic.
- Local tests → Local storage: verifies · Checks filesystem backend behavior.
- S3 tests → S3 storage: verifies · Checks S3 backend behavior.
- GCS tests → GCS storage: verifies · Checks GCS backend behavior.
- Manifest tests → Manifest: verifies · Checks append coordination behavior.
- Append tests → Manifest: covers · Exercises append behavior across backends.
- Append tests → Storage: covers · Validates append through the shared API.