# sanjeev0303/storage_ingestion architecture

> Architecture diagram of the GitHub repository sanjeev0303/storage_ingestion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sanjeev0303/storage_ingestion
- Repository: https://github.com/sanjeev0303/storage_ingestion

## Overview

This repository is an object-storage and data-governance platform: a web console and SDKs let users bring data in, organize and version datasets, validate and inspect them, then manage access, evidence, retention, and storage operations. The Go API is the backend entry point; the Python SDK also documents a local-folder ingestion workflow. The README is empty, so capability coverage comes from the supplied source index and homepage copy. Many frontend-to-backend and worker relationships are not shown in the excerpts and are omitted or marked unverified.

Main entry points: [`client/src/app/app`](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/app/app) (Web console), [`server/cmd/api/main.go`](https://github.com/sanjeev0303/storage_ingestion/blob/main/server/cmd/api/main.go) (Go API), [`sdk/python/osp/ingest.py`](https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/python/osp/ingest.py) (Folder ingestion), [`client/src/features/ingestion`](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/ingestion) (Ingestion workflow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["Web experience"]
  node_console["Web console"]
  node_ingestion["Ingestion workflow"]
  node_uploads["File uploads"]
  node_datasets["Datasets and versions"]
  node_catalog["Catalog"]
  node_validation["Validation and quality"]
  node_lineage["Lineage and evidence"]
  node_governance["Governance and audit"]
  node_streaming["Training data access"]
end

subgraph group_platform["Platform API"]
  node_api["Go API<br/>[main.go]"]
  node_postgres[("PostgreSQL repositories")]
end

subgraph group_sdk["Client SDKs"]
  node_python["Python SDK<br/>[client.py]"]
  node_python_ingest["Folder ingestion<br/>[ingest.py]"]
  node_branches["Branch operations<br/>[branches.py]"]
  node_go_agent["Edge agent<br/>[main.go]"]
end

subgraph group_operations["Storage operations"]
  node_storage["Object storage<br/>[object_storage.go]"]
  node_connections["Storage connections"]
  node_worker["Background workers<br/>[main.go]"]
end

node_user(("Platform user"))
node_provider["Storage providers"]

node_user -->|"uses"| node_console
node_console -.->|"starts"| node_ingestion
node_console -.->|"manages"| node_datasets
node_console -.->|"browses"| node_catalog
node_console -.->|"validates"| node_validation
node_console -.->|"inspects"| node_lineage
node_console -.->|"governs"| node_governance
node_console -.->|"configures"| node_streaming
node_ingestion -.->|"coordinates"| node_uploads
node_streaming -.->|"requests repacks"| node_api
node_python_ingest -->|"calls client"| node_python
node_branches -->|"calls client"| node_python
node_python_ingest -.->|"sends API requests"| node_api
node_go_agent -.->|"uploads data"| node_api
node_api -->|"opens database"| node_postgres
node_api -->|"configures storage"| node_storage
node_api -->|"routes connections"| node_connections
node_api -.->|"dispatches work"| node_worker
node_connections -.->|"tests connections"| node_provider
node_storage -.->|"stores objects"| node_provider

click node_console "https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/app/app"
click node_ingestion "https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/ingestion"
click node_uploads "https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/uploads"
click node_datasets "https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/datasets"
click node_catalog "https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/catalog"
click node_validation "https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/validation"
click node_lineage "https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/lineage"
click node_governance "https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/governance"
click node_streaming "https://github.com/sanjeev0303/storage_ingestion/blob/main/client/src/features/streaming/components/storage-layout-panel.tsx"
click node_python "https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/python/osp/client.py"
click node_python_ingest "https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/python/osp/ingest.py"
click node_branches "https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/python/osp/branches.py"
click node_go_agent "https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/go/cmd/osp-agent/main.go"
click node_api "https://github.com/sanjeev0303/storage_ingestion/blob/main/server/cmd/api/main.go"
click node_postgres "https://github.com/sanjeev0303/storage_ingestion/blob/main/server/internal/infrastructure/database/postgres/chunk_storage_repository.go"
click node_storage "https://github.com/sanjeev0303/storage_ingestion/blob/main/server/internal/infrastructure/storage/s3/object_storage.go"
click node_connections "https://github.com/sanjeev0303/storage_ingestion/blob/main/server/internal/handler/http/storage_connection_handler.go"
click node_worker "https://github.com/sanjeev0303/storage_ingestion/blob/main/server/cmd/upload-worker/main.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_console,node_ingestion,node_uploads,node_datasets,node_catalog,node_validation,node_lineage,node_governance,node_streaming,node_user toneBlue
class node_api,node_postgres,node_provider toneAmber
class node_python,node_python_ingest,node_branches,node_go_agent toneMint
class node_storage,node_connections,node_worker toneRose
```

## Components

### Web experience

- [Web console](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/app/app): `client/src/app/app`
- [Ingestion workflow](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/ingestion): `client/src/features/ingestion`
- [File uploads](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/uploads): `client/src/features/uploads`
- [Datasets and versions](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/datasets): `client/src/features/datasets`
- [Catalog](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/catalog): `client/src/features/catalog`
- [Validation and quality](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/validation): `client/src/features/validation`
- [Lineage and evidence](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/lineage): `client/src/features/lineage`
- [Governance and audit](https://github.com/sanjeev0303/storage_ingestion/tree/main/client/src/features/governance): `client/src/features/governance`
- [Training data access](https://github.com/sanjeev0303/storage_ingestion/blob/main/client/src/features/streaming/components/storage-layout-panel.tsx): `client/src/features/streaming/components/storage-layout-panel.tsx`

### Platform API

- [Go API](https://github.com/sanjeev0303/storage_ingestion/blob/main/server/cmd/api/main.go): `server/cmd/api/main.go`
- [PostgreSQL repositories](https://github.com/sanjeev0303/storage_ingestion/blob/main/server/internal/infrastructure/database/postgres/chunk_storage_repository.go): `server/internal/infrastructure/database/postgres/chunk_storage_repository.go`

### Client SDKs

- [Python SDK](https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/python/osp/client.py): `sdk/python/osp/client.py`
- [Folder ingestion](https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/python/osp/ingest.py): `sdk/python/osp/ingest.py`
- [Branch operations](https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/python/osp/branches.py): `sdk/python/osp/branches.py`
- [Edge agent](https://github.com/sanjeev0303/storage_ingestion/blob/main/sdk/go/cmd/osp-agent/main.go): `sdk/go/cmd/osp-agent/main.go`

### Storage operations

- [Object storage](https://github.com/sanjeev0303/storage_ingestion/blob/main/server/internal/infrastructure/storage/s3/object_storage.go): `server/internal/infrastructure/storage/s3/object_storage.go`
- [Storage connections](https://github.com/sanjeev0303/storage_ingestion/blob/main/server/internal/handler/http/storage_connection_handler.go): `server/internal/handler/http/storage_connection_handler.go`
- [Background workers](https://github.com/sanjeev0303/storage_ingestion/blob/main/server/cmd/upload-worker/main.go): `server/cmd/upload-worker/main.go`

### Other components

- **Platform user**
- **Storage providers**

## Connections

- Platform user → Web console: uses
- Web console → Ingestion workflow: starts
- Web console → Datasets and versions: manages
- Web console → Catalog: browses
- Web console → Validation and quality: validates
- Web console → Lineage and evidence: inspects
- Web console → Governance and audit: governs
- Web console → Training data access: configures
- Ingestion workflow → File uploads: coordinates
- Training data access → Go API: requests repacks
- Folder ingestion → Python SDK: calls client (evidence: `sdk/python/osp/ingest.py`)
- Branch operations → Python SDK: calls client (evidence: `sdk/python/osp/branches.py`)
- Folder ingestion → Go API: sends API requests
- Edge agent → Go API: uploads data
- Go API → PostgreSQL repositories: opens database (evidence: `server/cmd/api/main.go`)
- Go API → Object storage: configures storage (evidence: `server/cmd/api/main.go`)
- Go API → Storage connections: routes connections (evidence: `server/cmd/api/main.go`)
- Go API → Background workers: dispatches work
- Storage connections → Storage providers: tests connections
- Object storage → Storage providers: stores objects