# khaiphan29/logpulse architecture

> Architecture diagram of the GitHub repository khaiphan29/logpulse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/khaiphan29/logpulse
- Repository: https://github.com/khaiphan29/logpulse

## Overview

##### Project Type and Purpose
- This is a backend-focused, real-time log monitoring and analysis tool (“LogPulse”) built in Go.
- It ingests logs via Kafka, processes them, indexes them in Elasticsearch, and exposes a REST API for querying and alerting.
- It follows a message-driven, microservice-style architecture with distinct worker and API components.

##### File Structure
- Top-level “cmd” folder:
- cmd/api/main.go → the HTTP API service
- cmd/processor/main.go → the log-processing worker service
- cmd/create_es_indexes/main.go → one-off utility to set up ES indexes
- internal/ packages:
- internal/kafka → producer & consumer wrappers
- internal/elasticsearch → ES client and indexing logic
- internal/processor → core message-processing business logic
- internal/api → layered web server (router, handlers, auth, parsing, validation)
- internal/alert → alerting logic (error alerts, notifications)
- internal/config & internal/setup → service bootstrap and config loading
- configs & deployments:
- configs/kafka → Kafka broker property files
- deployments/docker-compose.yaml & k8s manifests → container orchestration
- pkg/ contains shared utilities (logging, errors), tests/ contains unit and integration tests.

##### README for Additional Insights
- Provides setup steps for Kafka brokers, topics, Elasticsearch, then running the services.
- Mentions live code reload (Air) and future features (search, analytics).
- No front-end component; primary user interface is the API.

##### Main Components
a. External Systems
- Kafka Cluster (one or more brokers, topics for raw logs and DLQs)
- Elasticsearch Cluster (indices for logs, DLQ data)
b. Services
- Log Processor Service
– Consumes from Kafka “raw-logs” topic
– Applies parsing, validation, enrichment
– Routes valid entries to ES, invalid to DLQ topics
- API Service
– Exposes REST endpoints for querying logs and alerts
– Reads from Elasticsearch
– Optionally publishes alert messages back to Kafka or triggers notifications
- ES-Index Setup Utility
– One-time or on-start tool to create ES index mappings
c. Supporting Components
- Config Loader & Service Bootstrap
- Shared Libraries (logging, error handling)
- Deployment Orchestrator (Docker-Compose / Kubernetes)

##### Relationships and Data Flows
- Processor Service:
→ Kafka Consumer from “raw-logs”
→ internal/processor logic → internal/elasticsearch client
→ Elasticsearch Index
→ Kafka Producer to “dlq” or “error-alerts” if needed
- API Service:
→ HTTP Client Requests → internal/api/router
→ Query Elasticsearch → Return results
→ Possibly produce alert messages to Kafka via internal/kafka/producer
- ES-Index Utility:
→ Reads configs → Calls ES APIs to set up index templates

##### Architectural Patterns and Design Principles
- Message-driven, event-streaming backbone (Kafka pub/sub)
- Worker pattern for parallel log processing
- Layered HTTP server (router → handlers → parsing/validation → service logic)
- Dependency injection via internal/setup & config module
- Single responsibility per binary (cmd/*)

##### Details
- Both normal and error paths (DLQ topics, error alerts)
- Bootstrap sequence: setup utility → create topics & indexes → start services
- Config management (configs/kafka vs. internal/config)
- If using Kubernetes, illustrate Deployments, Services, ConfigMaps, and how pods connect to Kafka and ES

## Diagram (Mermaid)

```mermaid
graph TD
  subgraph "Infrastructure"
    direction TB
    DataSources((Log Sources)):::infra
    Kafka[(Kafka Cluster)]:::infra
    DLQ[(logs-dlq topic)]:::infra
    ErrorAlerts[(error-alerts topic)]:::infra
    ESCluster[(Elasticsearch Cluster)]:::storage
  end

  subgraph "Log Processor Service"
    direction TB
    LP_Main["cmd/processor/main.go"]:::ingestion
    ProcLogic["internal/processor/processor.go"]:::ingestion
    KafkaCons["internal/kafka/consumer.go"]:::ingestion
    KafkaProdP["internal/kafka/producer.go"]:::ingestion
    ESClientP["internal/elasticsearch/elasticsearch.go"]:::ingestion
    ConfigP["internal/config/config.go"]:::ingestion
    SetupP1["internal/setup/service.go"]:::ingestion
  end

  subgraph "REST API Service"
    direction TB
    API_Main["cmd/api/main.go"]:::api
    Router["internal/api/router/router.go"]:::api
    LogHandler["internal/api/handlers/log_handler.go"]:::api
    Auth["internal/api/auth/auth.go"]:::api
    Parser["internal/api/parsing/parser.go"]:::api
    Validation["internal/api/validation/validation.go"]:::api
    ESClientA["internal/elasticsearch/elasticsearch.go"]:::api
    KafkaProdA["internal/kafka/producer.go"]:::api
    ConfigA["internal/config/config.go"]:::api
    SetupP2["internal/setup/service.go"]:::api
  end

  subgraph "ES-Index Setup Utility"
    direction TB
    ESU_Main["cmd/create_es_indexes/main.go"]:::cli
    ESU_ESClient["internal/elasticsearch/elasticsearch.go"]:::cli
    ESU_Config["internal/config/config.go"]:::cli
    ESU_Setup["internal/setup/service.go"]:::cli
  end

  subgraph "Shared Utilities"
    direction TB
    Logger["pkg/logger/logger.go"]:::shared
    Errors["pkg/errors/errors.go"]:::shared
    AlertComp["internal/alert/error_alert.go"]:::shared
  end

  DataSources -->|produce logs| Kafka
  Kafka -->|raw-logs| KafkaCons
  KafkaCons --> ProcLogic
  ProcLogic --> ESClientP
  ESClientP --> ESCluster
  ProcLogic --> KafkaProdP
  KafkaProdP --> DLQ
  ProcLogic --> AlertComp

  API_Main --> Router
  Router --> LogHandler
  LogHandler --> ESClientA
  ESClientA --> ESCluster
  API_Main --> KafkaProdA

  ESU_Main --> ESU_ESClient
  ESU_ESClient --> ESCluster

  ConfigP --> LP_Main
  SetupP1 --> LP_Main
  ConfigA --> API_Main
  SetupP2 --> API_Main
  ESU_Config --> ESU_Main
  ESU_Setup --> ESU_Main

  Logger --> LP_Main
  Logger --> API_Main

  click LP_Main "https://github.com/khaiphan29/logpulse/blob/main/cmd/processor/main.go"
  click ProcLogic "https://github.com/khaiphan29/logpulse/blob/main/internal/processor/processor.go"
  click KafkaCons "https://github.com/khaiphan29/logpulse/blob/main/internal/kafka/consumer.go"
  click KafkaProdP "https://github.com/khaiphan29/logpulse/blob/main/internal/kafka/producer.go"
  click ESClientP "https://github.com/khaiphan29/logpulse/blob/main/internal/elasticsearch/elasticsearch.go"
  click ConfigP "https://github.com/khaiphan29/logpulse/blob/main/internal/config/config.go"
  click SetupP1 "https://github.com/khaiphan29/logpulse/blob/main/internal/setup/service.go"

  click API_Main "https://github.com/khaiphan29/logpulse/blob/main/cmd/api/main.go"
  click Router "https://github.com/khaiphan29/logpulse/blob/main/internal/api/router/router.go"
  click LogHandler "https://github.com/khaiphan29/logpulse/blob/main/internal/api/handlers/log_handler.go"
  click Auth "https://github.com/khaiphan29/logpulse/blob/main/internal/api/auth/auth.go"
  click Parser "https://github.com/khaiphan29/logpulse/blob/main/internal/api/parsing/parser.go"
  click Validation "https://github.com/khaiphan29/logpulse/blob/main/internal/api/validation/validation.go"
  click ESClientA "https://github.com/khaiphan29/logpulse/blob/main/internal/elasticsearch/elasticsearch.go"
  click KafkaProdA "https://github.com/khaiphan29/logpulse/blob/main/internal/kafka/producer.go"
  click ConfigA "https://github.com/khaiphan29/logpulse/blob/main/internal/config/config.go"
  click SetupP2 "https://github.com/khaiphan29/logpulse/blob/main/internal/setup/service.go"

  click ESU_Main "https://github.com/khaiphan29/logpulse/blob/main/cmd/create_es_indexes/main.go"
  click ESU_ESClient "https://github.com/khaiphan29/logpulse/blob/main/internal/elasticsearch/elasticsearch.go"
  click ESU_Config "https://github.com/khaiphan29/logpulse/blob/main/internal/config/config.go"
  click ESU_Setup "https://github.com/khaiphan29/logpulse/blob/main/internal/setup/service.go"

  classDef ingestion stroke:#3498db,fill:#D6EAF8
  classDef api stroke:#27AE60,fill:#D5F5E3
  classDef storage stroke:#F1C40F,fill:#FCF3CF
  classDef infra stroke:#7F8C8D,fill:#D5D8DC
  classDef cli stroke:#E74C3C,fill:#FADBD8
  classDef shared stroke:#8E44AD,fill:#E8DAEF
```
