How sanjeev0303/storage_ingestion works
This repository is an object-storage and data-governance platform: a web console and SDKs let users bring data in, organize and version datasets, validate and inspect them, then manage access, evidence, retention, and storage operations. The Go API is the backend entry point; the Python SDK also documents a local-folder ingestion workflow. The README is empty, so capability coverage comes from the supplied source index and homepage copy. Many frontend-to-backend and worker relationships are not shown in the excerpts and are omitted or marked unverified.
Main entry points: app Web console · main.go Go API · ingest.py Folder ingestion · ingestion Ingestion workflow
Components 20
Web experience
- Web console
client/src/app/app - Ingestion workflow
client/src/features/ingestion - File uploads
client/src/features/uploads - Datasets and versions
client/src/features/datasets - Catalog
client/src/features/catalog - Validation and quality
client/src/features/validation - Lineage and evidence
client/src/features/lineage - Governance and audit
client/src/features/governance - Training data access
client/src/features/streaming/components/storage-layout-panel.tsx
Platform API
- Go API
server/cmd/api/main.go - PostgreSQL repositories
server/internal/infrastructure/database/postgres/chunk_storage_repository.go
Client SDKs
- Python SDK
sdk/python/osp/client.py - Folder ingestion
sdk/python/osp/ingest.py - Branch operations
sdk/python/osp/branches.py - Edge agent
sdk/go/cmd/osp-agent/main.go
Storage operations
- Object storage
server/internal/infrastructure/storage/s3/object_storage.go - Storage connections
server/internal/handler/http/storage_connection_handler.go - Background workers
server/cmd/upload-worker/main.go
Other components
- Platform user
- Storage providers
Connections 20
- Platform user → Web console: uses
- Web console → Ingestion workflow: starts
- Web console → Datasets and versions: manages
- Web console → Catalog: browses
- Web console → Validation and quality: validates
- Web console → Lineage and evidence: inspects
- Web console → Governance and audit: governs
- Web console → Training data access: configures
- Ingestion workflow → File uploads: coordinates
- Training data access → Go API: requests repacks
- Folder ingestion → Python SDK: calls client
sdk/python/osp/ingest.py - Branch operations → Python SDK: calls client
sdk/python/osp/branches.py - Folder ingestion → Go API: sends API requests
- Edge agent → Go API: uploads data
- Go API → PostgreSQL repositories: opens database
server/cmd/api/main.go - Go API → Object storage: configures storage
server/cmd/api/main.go - Go API → Storage connections: routes connections
server/cmd/api/main.go - Go API → Background workers: dispatches work
- Storage connections → Storage providers: tests connections
- Object storage → Storage providers: stores objects