# pawapotako/go-data-loader-api architecture

> Architecture diagram of the GitHub repository pawapotako/go-data-loader-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pawapotako/go-data-loader-api
- Repository: https://github.com/pawapotako/go-data-loader-api

## Overview

This is a Go API for loading pipe-delimited files into PostgreSQL. The principal workflow is an HTTP load request, checkpoint lookup, file streaming and parsing, batched concurrent inserts, per-record error capture on batch failure, and checkpoint update with a load summary. The README documents these stages; sampled source confirms the API wiring and use-case calls. Repository internals were not excerpted, so storage connections are shown only where the use case invokes repository methods.

Main entry points: [`modules/data_loader/usecases/data_loader_usecases.go`](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/usecases/data_loader_usecases.go) (Load orchestration), [`modules/servers/handler.go`](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/servers/handler.go) (Route wiring), [`modules/data_loader/controllers/data_loader_controllers.go`](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/controllers/data_loader_controllers.go) (Load controller), [`pkg/databases/postgresql.go`](https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/databases/postgresql.go) (PostgreSQL)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["HTTP API"]
  node_server["HTTP server<br/>[server.go]"]
  node_routing["Route wiring<br/>[handler.go]"]
  node_controller["Load controller"]
  node_app_entry["Application entry<br/>[main.go]"]
end

subgraph group_loader["File Loading"]
  node_usecase["Load orchestration"]
  node_file_reader["Line streaming<br/>[reader.go]"]
  node_parser["Pipe parsing<br/>[string.go]"]
  node_batch_builder["Batch builder<br/>[builder.go]"]
  node_checkpoint_tracker["Checkpoint tracker<br/>[tracker.go]"]
  node_load_request["Load request<br/>[data_loader.go]"]
  node_data_rows["Data records<br/>[data_entities.go]"]
  node_worker_result["Worker result<br/>[data_loader.go]"]
end

subgraph group_persistence["Persistence"]
  node_checkpoint_entity["Checkpoint record"]
  node_error_entity["Load error record"]
  node_data_repo["Data repository"]
  node_checkpoint_repo["Checkpoint repository"]
  node_error_repo["Error repository"]
  node_postgres[("PostgreSQL<br/>[postgresql.go]")]
end

node_autosys(("AutoSys / caller"))

node_autosys -->|"submits request"| node_controller
node_server -->|"maps routes"| node_routing
node_routing -->|"registers route"| node_controller
node_controller -->|"calls load"| node_usecase
node_controller -->|"parses into"| node_load_request
node_usecase -->|"gets and updates"| node_checkpoint_repo
node_usecase -->|"streams file"| node_file_reader
node_usecase -->|"parses lines"| node_parser
node_usecase -->|"builds batches"| node_batch_builder
node_usecase -->|"tracks progress"| node_checkpoint_tracker
node_usecase -->|"inserts records"| node_data_repo
node_usecase -->|"saves failures"| node_error_repo
node_usecase -->|"creates records"| node_data_rows
node_usecase -->|"collects results"| node_worker_result
node_usecase -->|"records error details"| node_error_entity
node_app_entry -->|"opens connection"| node_postgres
node_app_entry -->|"starts server"| node_server
node_routing -->|"constructs"| node_data_repo
node_routing -->|"constructs"| node_checkpoint_repo
node_routing -->|"constructs"| node_error_repo
node_routing -->|"constructs"| node_usecase
node_usecase -.->|"uses checkpoint"| node_checkpoint_entity
node_usecase -->|"persists details"| node_error_entity
node_data_repo -.->|"uses database"| node_postgres
node_checkpoint_repo -.->|"uses database"| node_postgres
node_error_repo -.->|"uses database"| node_postgres

click node_server "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/servers/server.go"
click node_routing "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/servers/handler.go"
click node_controller "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/controllers/data_loader_controllers.go"
click node_usecase "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/usecases/data_loader_usecases.go"
click node_file_reader "https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/files/reader.go"
click node_parser "https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/utils/string.go"
click node_batch_builder "https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/batches/builder.go"
click node_checkpoint_tracker "https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/batches/tracker.go"
click node_load_request "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/data_loader.go"
click node_data_rows "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/data_entities.go"
click node_worker_result "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/data_loader.go"
click node_checkpoint_entity "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/checkpoint_entities.go"
click node_error_entity "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/data_load_error_entities.go"
click node_data_repo "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/repositories/data_repositories.go"
click node_checkpoint_repo "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/repositories/checkpoint_repositories.go"
click node_error_repo "https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/repositories/data_load_error_repositories.go"
click node_postgres "https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/databases/postgresql.go"
click node_app_entry "https://github.com/pawapotako/go-data-loader-api/blob/master/app/main.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_routing,node_controller,node_app_entry toneBlue
class node_usecase,node_file_reader,node_parser,node_batch_builder,node_checkpoint_tracker,node_load_request,node_data_rows,node_worker_result toneAmber
class node_checkpoint_entity,node_error_entity,node_data_repo,node_checkpoint_repo,node_error_repo,node_postgres toneMint
class node_autosys toneIndigo
```

## Components

### HTTP API

- [HTTP server](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/servers/server.go): `modules/servers/server.go`
- [Route wiring](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/servers/handler.go): `modules/servers/handler.go`
- [Load controller](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/controllers/data_loader_controllers.go): `modules/data_loader/controllers/data_loader_controllers.go`
- [Application entry](https://github.com/pawapotako/go-data-loader-api/blob/master/app/main.go): `app/main.go`

### File Loading

- [Load orchestration](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/usecases/data_loader_usecases.go): `modules/data_loader/usecases/data_loader_usecases.go`
- [Line streaming](https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/files/reader.go): `pkg/files/reader.go`
- [Pipe parsing](https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/utils/string.go): `pkg/utils/string.go`
- [Batch builder](https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/batches/builder.go): `pkg/batches/builder.go`
- [Checkpoint tracker](https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/batches/tracker.go): `pkg/batches/tracker.go`
- [Load request](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/data_loader.go): `modules/entities/data_loader.go`
- [Data records](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/data_entities.go): `modules/entities/data_entities.go`
- [Worker result](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/data_loader.go): `modules/entities/data_loader.go`

### Persistence

- [Checkpoint record](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/checkpoint_entities.go): `modules/entities/checkpoint_entities.go`
- [Load error record](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/entities/data_load_error_entities.go): `modules/entities/data_load_error_entities.go`
- [Data repository](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/repositories/data_repositories.go): `modules/data_loader/repositories/data_repositories.go`
- [Checkpoint repository](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/repositories/checkpoint_repositories.go): `modules/data_loader/repositories/checkpoint_repositories.go`
- [Error repository](https://github.com/pawapotako/go-data-loader-api/blob/master/modules/data_loader/repositories/data_load_error_repositories.go): `modules/data_loader/repositories/data_load_error_repositories.go`
- [PostgreSQL](https://github.com/pawapotako/go-data-loader-api/blob/master/pkg/databases/postgresql.go): `pkg/databases/postgresql.go`

### Other components

- **AutoSys / caller**

## Connections

- AutoSys / caller → Load controller: submits request (evidence: `README.md`)
- HTTP server → Route wiring: maps routes (evidence: `modules/servers/server.go`)
- Route wiring → Load controller: registers route (evidence: `modules/servers/handler.go`)
- Load controller → Load orchestration: calls load (evidence: `modules/data_loader/controllers/data_loader_controllers.go`)
- Load controller → Load request: parses into (evidence: `modules/data_loader/controllers/data_loader_controllers.go`)
- Load orchestration → Checkpoint repository: gets and updates (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Line streaming: streams file (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Pipe parsing: parses lines (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Batch builder: builds batches (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Checkpoint tracker: tracks progress (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Data repository: inserts records (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Error repository: saves failures (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Data records: creates records (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Worker result: collects results (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Load orchestration → Load error record: records error details (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Application entry → PostgreSQL: opens connection (evidence: `app/main.go`)
- Application entry → HTTP server: starts server (evidence: `app/main.go`)
- Route wiring → Data repository: constructs (evidence: `modules/servers/handler.go`)
- Route wiring → Checkpoint repository: constructs (evidence: `modules/servers/handler.go`)
- Route wiring → Error repository: constructs (evidence: `modules/servers/handler.go`)
- Route wiring → Load orchestration: constructs (evidence: `modules/servers/handler.go`)
- Load orchestration → Checkpoint record: uses checkpoint
- Load orchestration → Load error record: persists details (evidence: `modules/data_loader/usecases/data_loader_usecases.go`)
- Data repository → PostgreSQL: uses database
- Checkpoint repository → PostgreSQL: uses database
- Error repository → PostgreSQL: uses database
