# krishnazawar/distributed-logger architecture

> Architecture diagram of the GitHub repository krishnazawar/distributed-logger, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/krishnazawar/distributed-logger
- Repository: https://github.com/krishnazawar/distributed-logger

## Overview

The repository implements a learning-oriented distributed logging pipeline: applications emit structured logs through the SDK, an agent tails configured files and ships batches, and an ingestion service enriches and persists them. The README also describes retries, buffering, and durable storage; sampled source confirms file-based offsets and log persistence, but not every documented guarantee. The queue is explicitly optional and not implemented, so it is omitted. Agent and ingestion internals are represented only where source or README evidence supports them.

Main entry points: [`logging-agent/internal/entity/collector.go`](https://github.com/krishnazawar/distributed-logger/blob/master/logging-agent/internal/entity/collector.go) (Collector and shipper), [`ingestion-node/internal/handler/handlers.go`](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/handler/handlers.go) (Validate and enrich), [`ingestion-node/internal/logwriter/logWriter.go`](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/logwriter/logWriter.go) (File writer), [`ingestion-node/internal/router/router.go`](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/router/router.go) (HTTP routes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sdk["Logging SDK"]
  node_sdk_logger["Logger API<br/>[logger.go]"]
  node_sdk_event["Structured events<br/>[logEvent.go]"]
  node_sdk_writer["Output writer<br/>[logWriter.go]"]
end

subgraph group_agent["Log Shipping"]
  node_agent_main["Agent startup<br/>[main.go]"]
  node_agent["Collector and shipper<br/>[collector.go]"]
  node_agent_config["Agent configuration<br/>[config.go]"]
  node_offsets["Read offsets"]
end

subgraph group_ingestion["Ingestion Service"]
  node_ingest_main["Ingestion startup<br/>[main.go]"]
  node_router["HTTP routes<br/>[router.go]"]
  node_handler["Validate and enrich<br/>[handlers.go]"]
  node_log_model["Log records<br/>[models.go]"]
  node_ingest_config["Storage configuration<br/>[config.go]"]
end

subgraph group_storage["Persistence"]
  node_logwriter["File writer<br/>[logWriter.go]"]
  node_log_store["Persisted log file"]
end

node_service(("Application service"))
node_local_logs["Local log files"]
node_http(("HTTP transport"))

node_service -->|"emits logs"| node_sdk_logger
node_sdk_logger -->|"creates events"| node_sdk_event
node_sdk_event -->|"writes events"| node_sdk_writer
node_sdk_writer -->|"writes locally"| node_local_logs
node_agent_main -->|"starts reading"| node_agent
node_agent -->|"loads settings"| node_agent_config
node_agent -->|"tails files"| node_local_logs
node_agent -->|"loads and saves"| node_offsets
node_agent -->|"posts batches"| node_http
node_http -->|"routes requests"| node_router
node_ingest_main -->|"registers routes"| node_router
node_router -->|"dispatches ingest"| node_handler
node_handler -->|"parses records"| node_log_model
node_handler -->|"persists enriched logs"| node_logwriter
node_logwriter -->|"loads destination"| node_ingest_config
node_logwriter -->|"appends records"| node_log_store

click node_sdk_logger "https://github.com/krishnazawar/distributed-logger/blob/master/logger-sdk/logger.go"
click node_sdk_event "https://github.com/krishnazawar/distributed-logger/blob/master/logger-sdk/logEvent.go"
click node_sdk_writer "https://github.com/krishnazawar/distributed-logger/blob/master/logger-sdk/logWriter.go"
click node_agent_main "https://github.com/krishnazawar/distributed-logger/blob/master/logging-agent/cmd/main.go"
click node_agent "https://github.com/krishnazawar/distributed-logger/blob/master/logging-agent/internal/entity/collector.go"
click node_agent_config "https://github.com/krishnazawar/distributed-logger/blob/master/logging-agent/internal/config/config.go"
click node_ingest_main "https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/cmd/main.go"
click node_router "https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/router/router.go"
click node_handler "https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/handler/handlers.go"
click node_log_model "https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/entity/models.go"
click node_ingest_config "https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/config/config.go"
click node_logwriter "https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/logwriter/logWriter.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sdk_logger,node_sdk_event,node_sdk_writer toneBlue
class node_agent_main,node_agent,node_agent_config,node_offsets toneAmber
class node_ingest_main,node_router,node_handler,node_log_model,node_ingest_config toneMint
class node_logwriter,node_log_store toneRose
class node_service,node_local_logs,node_http toneIndigo
```

## Components

### Logging SDK

- [Logger API](https://github.com/krishnazawar/distributed-logger/blob/master/logger-sdk/logger.go): `logger-sdk/logger.go`
- [Structured events](https://github.com/krishnazawar/distributed-logger/blob/master/logger-sdk/logEvent.go): `logger-sdk/logEvent.go`
- [Output writer](https://github.com/krishnazawar/distributed-logger/blob/master/logger-sdk/logWriter.go): `logger-sdk/logWriter.go`

### Log Shipping

- [Agent startup](https://github.com/krishnazawar/distributed-logger/blob/master/logging-agent/cmd/main.go): `logging-agent/cmd/main.go`
- [Collector and shipper](https://github.com/krishnazawar/distributed-logger/blob/master/logging-agent/internal/entity/collector.go): `logging-agent/internal/entity/collector.go`
- [Agent configuration](https://github.com/krishnazawar/distributed-logger/blob/master/logging-agent/internal/config/config.go): `logging-agent/internal/config/config.go`
- **Read offsets**

### Ingestion Service

- [Ingestion startup](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/cmd/main.go): `ingestion-node/cmd/main.go`
- [HTTP routes](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/router/router.go): `ingestion-node/internal/router/router.go`
- [Validate and enrich](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/handler/handlers.go): `ingestion-node/internal/handler/handlers.go`
- [Log records](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/entity/models.go): `ingestion-node/internal/entity/models.go`
- [Storage configuration](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/config/config.go): `ingestion-node/internal/config/config.go`

### Persistence

- [File writer](https://github.com/krishnazawar/distributed-logger/blob/master/ingestion-node/internal/logwriter/logWriter.go): `ingestion-node/internal/logwriter/logWriter.go`
- **Persisted log file**

### Other components

- **Application service**
- **Local log files**
- **HTTP transport**

## Connections

- Application service → Logger API: emits logs
- Logger API → Structured events: creates events
- Structured events → Output writer: writes events
- Output writer → Local log files: writes locally
- Agent startup → Collector and shipper: starts reading
- Collector and shipper → Agent configuration: loads settings
- Collector and shipper → Local log files: tails files
- Collector and shipper → Read offsets: loads and saves
- Collector and shipper → HTTP transport: posts batches
- HTTP transport → HTTP routes: routes requests
- Ingestion startup → HTTP routes: registers routes
- HTTP routes → Validate and enrich: dispatches ingest
- Validate and enrich → Log records: parses records
- Validate and enrich → File writer: persists enriched logs
- File writer → Storage configuration: loads destination
- File writer → Persisted log file: appends records