How vindyang/cs464-project works
1. This repository is a full-stack distributed file-storage system: a Go backend split into multiple services, a Next.js/Bun frontend, and Docker Compose wiring that runs the whole stack from the repo root.
2. The main request path starts at the frontend and the gateway. The frontend/ app is the user-facing UI, while backend/services/gateway/ is the HTTP edge layer, implemented with Nginx config plus OpenAPI/Swagger assets. The gateway is the public backend entrypoint and is the boundary between browser traffic and internal services.
3. The backend is organized as service-oriented components rather than one monolith. Each service has its own cmd/, internal/, Dockerfile, and tests, which makes the service boundaries explicit and independently runnable.
4. orchestrator is the workflow coordinator. Its job is to execute higher-level file operations by calling other backend services through shared HTTP clients. In the architecture diagram, it should sit in the middle of upload/download flows, not as a data store.
5. sharding is the service that determines how files are split or distributed. It exposes an API and has a small application layer, suggesting it is responsible for shard sizing/placement logic rather than persistence.
6. shardmap is the metadata authority. It contains handlers, application logic, and repository implementations for files and shards, so it is the service that persists and serves the mapping between logical files, their shards, and the storage nodes/providers that hold them.
7. adapter is the execution layer that talks to external storage providers. Its handlers and file_operations_service indicate it performs the actual file I/O work, while shared adapter code includes Google Drive and S3 integrations plus OAuth handling for Drive auth. This is the boundary where backend logic leaves the system and reaches third-party storage APIs.
8. The shared package under backend/services/shared/ is a cross-cutting code layer, not a separate runtime. It holds DTOs, middleware, transport helpers, database access, reusable clients for service-to-service calls, common models/types, and provider adapters. That layer defines the contracts between services and keeps duplication down.
9. Data persistence is centralized around the shard-mapping side. The repo includes schema.sql, seed.sql, and docker/init-local.sql, plus repository code inside shardmap, which indicates an SQL-backed metadata store initialized locally for development and testing.
10. The frontend is a rich authenticated admin console. Its route groups include dashboard, nodes, providers, and settings, with reusable components for file tables, upload dialogs, shard progress, system health, and provider cards. It consumes backend APIs through frontend/lib/api/ and keeps UI state in dedicated stores.
11. The frontend also supports local mocked data paths. frontend/lib/mocks/ and store modules suggest the UI can be developed or exercised without the live backend, which is useful for isolated UI work.
12. Testing is layered by service. Each backend service has unit tests, backend/tests/integration/ contains orchestrator contract/failure-path tests, and backend/tests/e2e/ provides end-to-end coverage. There are also local test utilities like cmd/mockserver and cmd/test-db for dependency simulation.
13. Deployment is container-first. Docker Compose profiles separate backend and full runs, so the backend services, gateway, and frontend are intended to be composed together as a system, with the gateway sitting at the ingress and the other services forming the internal control plane and storage workflow.
Main entry points: docker-compose.yml Compose · shared Shared Code · orchestrator Orchestrator · gateway Gateway
Components 15
Frontend
Next.js/Bun admin console and UI state
- Admin Console · Authenticated browser UI for dashboard, nodes, providers, and settings
frontend/app - API Client · Typed client layer used by the UI to call backend APIs
frontend/lib/api - UI State · Client-side stores and mock data paths for local UI workflows
frontend/lib/store
Backend Services
Go runtime services for ingress, workflow, sharding, metadata, and storage execution
- Gateway · Public ingress layer with Nginx config and OpenAPI/Swagger assets
backend/services/gateway - Orchestrator · Coordinates upload and download workflows by calling backend services
backend/services/orchestrator - Sharding · Determines shard sizing and distribution logic through its API and app layer
backend/services/sharding - Shard Map · Persists and serves file, shard, and storage-node mappings
backend/services/shardmap - Adapter · Executes file operations against external storage providers such as Drive and S3
backend/services/adapter - SQL Store · Schema and seed SQL backing the shard-mapping and workflow metadata
backend/schema.sql
Shared Layer
Cross-cutting DTOs, clients, transport, auth, and provider adapters
- Shared Code · Reusable DTOs, middleware, clients, transport helpers, models, and provider adapters
backend/services/shared
Tests
Integration and end-to-end coverage plus local test harnesses
- Integration Tests · Orchestrator contract and failure-path tests for service interaction
backend/tests/integration - E2E Tests · End-to-end coverage for the composed backend system
backend/tests/e2e - Mock Server · Local dependency simulator for backend testing
backend/cmd/mockserver - Test DB · Local database bootstrap utility for tests and development
backend/cmd/test-db
Deploy
Repo-level container orchestration and initialization
- Compose · Container-first system assembly for backend and full-stack runs
docker-compose.yml
Connections 26
- Admin Console → API Client: reads data from · The UI uses the API client layer for backend communication
- Admin Console → UI State: uses state · The UI relies on local stores and mock paths for interaction state
- API Client → Gateway: calls · Browser traffic is routed through the public gateway
- Gateway → Orchestrator: routes workflow · The edge layer forwards core file workflows to the orchestrator
- Gateway → Shard Map: exposes metadata · Gateway-backed APIs surface shard and file metadata operations
- Orchestrator → Sharding: plans shards · Workflow coordination depends on sharding decisions
- Orchestrator → Shard Map: updates metadata · Workflow execution reads and writes shard-map state
- Orchestrator → Adapter: executes storage ops · The orchestrator drives storage actions through the adapter service
- Sharding → Shared Code: uses contracts · Sharding shares DTOs and transport helpers from the common layer
- Shard Map → SQL Store: persists to · Shard mapping service stores file and shard metadata in SQL
- Adapter → Shared Code: uses providers · Adapter code reuses shared provider integrations and OAuth helpers
- Shared Code → Gateway: shared transport · Common middleware and transport helpers support service boundaries
- Shared Code → Orchestrator: shared clients · Orchestrator uses shared HTTP clients and DTOs
- Shared Code → Sharding: shared clients · Sharding uses common contracts and transport helpers
- Shared Code → Shard Map: shared contracts · Shard-map service depends on common models and middleware
- Shared Code → Adapter: shared adapters · Adapter service reuses common provider and transport code
- Integration Tests → Orchestrator: verifies · Contract tests focus on orchestrator behavior and failure handling
- E2E Tests → Compose: runs against · End-to-end tests target the composed stack
- Mock Server → Integration Tests: supports · Mock services simulate backend dependencies for tests
- Test DB → Integration Tests: supports · Local test database bootstrap supports integration scenarios
- Compose → Gateway: starts · Compose wires the public ingress and internal services together
- Compose → Orchestrator: starts · Compose includes the workflow coordinator service
- Compose → Sharding: starts · Compose includes the shard decision service
- Compose → Shard Map: starts · Compose includes the metadata authority service
- Compose → Adapter: starts · Compose includes the storage execution service
- Compose → Admin Console: starts · Compose includes the browser UI for full-stack runs