How pawapotako/go-data-loader-api works
This is a Go API for loading pipe-delimited files into PostgreSQL. The principal workflow is an HTTP load request, checkpoint lookup, file streaming and parsing, batched concurrent inserts, per-record error capture on batch failure, and checkpoint update with a load summary. The README documents these stages; sampled source confirms the API wiring and use-case calls. Repository internals were not excerpted, so storage connections are shown only where the use case invokes repository methods.
Main entry points: data_loader_usecases.go Load orchestration · handler.go Route wiring · data_loader_controllers.go Load controller · postgresql.go PostgreSQL
Components 19
HTTP API
- HTTP server
modules/servers/server.go - Route wiring
modules/servers/handler.go - Load controller
modules/data_loader/controllers/data_loader_controllers.go - Application entry
app/main.go
File Loading
- Load orchestration
modules/data_loader/usecases/data_loader_usecases.go - Line streaming
pkg/files/reader.go - Pipe parsing
pkg/utils/string.go - Batch builder
pkg/batches/builder.go - Checkpoint tracker
pkg/batches/tracker.go - Load request
modules/entities/data_loader.go - Data records
modules/entities/data_entities.go - Worker result
modules/entities/data_loader.go
Persistence
- Checkpoint record
modules/entities/checkpoint_entities.go - Load error record
modules/entities/data_load_error_entities.go - Data repository
modules/data_loader/repositories/data_repositories.go - Checkpoint repository
modules/data_loader/repositories/checkpoint_repositories.go - Error repository
modules/data_loader/repositories/data_load_error_repositories.go - PostgreSQL
pkg/databases/postgresql.go
Other components
- AutoSys / caller
Connections 26
- AutoSys / caller → Load controller: submits request
README.md - HTTP server → Route wiring: maps routes
modules/servers/server.go - Route wiring → Load controller: registers route
modules/servers/handler.go - Load controller → Load orchestration: calls load
modules/data_loader/controllers/data_loader_controllers.go - Load controller → Load request: parses into
modules/data_loader/controllers/data_loader_controllers.go - Load orchestration → Checkpoint repository: gets and updates
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Line streaming: streams file
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Pipe parsing: parses lines
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Batch builder: builds batches
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Checkpoint tracker: tracks progress
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Data repository: inserts records
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Error repository: saves failures
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Data records: creates records
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Worker result: collects results
modules/data_loader/usecases/data_loader_usecases.go - Load orchestration → Load error record: records error details
modules/data_loader/usecases/data_loader_usecases.go - Application entry → PostgreSQL: opens connection
app/main.go - Application entry → HTTP server: starts server
app/main.go - Route wiring → Data repository: constructs
modules/servers/handler.go - Route wiring → Checkpoint repository: constructs
modules/servers/handler.go - Route wiring → Error repository: constructs
modules/servers/handler.go - Route wiring → Load orchestration: constructs
modules/servers/handler.go - Load orchestration → Checkpoint record: uses checkpoint
- Load orchestration → Load error record: persists details
modules/data_loader/usecases/data_loader_usecases.go - Data repository → PostgreSQL: uses database
- Checkpoint repository → PostgreSQL: uses database
- Error repository → PostgreSQL: uses database