How krishnazawar/distributed-logger works
The repository implements a learning-oriented distributed logging pipeline: applications emit structured logs through the SDK, an agent tails configured files and ships batches, and an ingestion service enriches and persists them. The README also describes retries, buffering, and durable storage; sampled source confirms file-based offsets and log persistence, but not every documented guarantee. The queue is explicitly optional and not implemented, so it is omitted. Agent and ingestion internals are represented only where source or README evidence supports them.
Main entry points: collector.go Collector and shipper · handlers.go Validate and enrich · logWriter.go File writer · router.go HTTP routes
Components 17
Logging SDK
- Logger API
logger-sdk/logger.go - Structured events
logger-sdk/logEvent.go - Output writer
logger-sdk/logWriter.go
Log Shipping
- Agent startup
logging-agent/cmd/main.go - Collector and shipper
logging-agent/internal/entity/collector.go - Agent configuration
logging-agent/internal/config/config.go - Read offsets
Ingestion Service
- Ingestion startup
ingestion-node/cmd/main.go - HTTP routes
ingestion-node/internal/router/router.go - Validate and enrich
ingestion-node/internal/handler/handlers.go - Log records
ingestion-node/internal/entity/models.go - Storage configuration
ingestion-node/internal/config/config.go
Persistence
- File writer
ingestion-node/internal/logwriter/logWriter.go - Persisted log file
Other components
- Application service
- Local log files
- HTTP transport
Connections 16
- Application service → Logger API: emits logs
- Logger API → Structured events: creates events
- Structured events → Output writer: writes events
- Output writer → Local log files: writes locally
- Agent startup → Collector and shipper: starts reading
- Collector and shipper → Agent configuration: loads settings
- Collector and shipper → Local log files: tails files
- Collector and shipper → Read offsets: loads and saves
- Collector and shipper → HTTP transport: posts batches
- HTTP transport → HTTP routes: routes requests
- Ingestion startup → HTTP routes: registers routes
- HTTP routes → Validate and enrich: dispatches ingest
- Validate and enrich → Log records: parses records
- Validate and enrich → File writer: persists enriched logs
- File writer → Storage configuration: loads destination
- File writer → Persisted log file: appends records