How khaiphan29/logpulse works
Project Type and Purpose
• This is a backend-focused, real-time log monitoring and analysis tool (“LogPulse”) built in Go.
• It ingests logs via Kafka, processes them, indexes them in Elasticsearch, and exposes a REST API for querying and alerting.
• It follows a message-driven, microservice-style architecture with distinct worker and API components.
File Structure
• Top-level “cmd” folder:
• cmd/api/main.go → the HTTP API service
• cmd/processor/main.go → the log-processing worker service
• cmd/create_es_indexes/main.go → one-off utility to set up ES indexes
• internal/ packages:
• internal/kafka → producer & consumer wrappers
• internal/elasticsearch → ES client and indexing logic
• internal/processor → core message-processing business logic
• internal/api → layered web server (router, handlers, auth, parsing, validation)
• internal/alert → alerting logic (error alerts, notifications)
• internal/config & internal/setup → service bootstrap and config loading
• configs & deployments:
• configs/kafka → Kafka broker property files
• deployments/docker-compose.yaml & k8s manifests → container orchestration
• pkg/ contains shared utilities (logging, errors), tests/ contains unit and integration tests.
README for Additional Insights
• Provides setup steps for Kafka brokers, topics, Elasticsearch, then running the services.
• Mentions live code reload (Air) and future features (search, analytics).
• No front-end component; primary user interface is the API.
Main Components
a. External Systems
• Kafka Cluster (one or more brokers, topics for raw logs and DLQs)
• Elasticsearch Cluster (indices for logs, DLQ data)
b. Services
• Log Processor Service
– Consumes from Kafka “raw-logs” topic
– Applies parsing, validation, enrichment
– Routes valid entries to ES, invalid to DLQ topics
• API Service
– Exposes REST endpoints for querying logs and alerts
– Reads from Elasticsearch
– Optionally publishes alert messages back to Kafka or triggers notifications
• ES-Index Setup Utility
– One-time or on-start tool to create ES index mappings
c. Supporting Components
• Config Loader & Service Bootstrap
• Shared Libraries (logging, error handling)
• Deployment Orchestrator (Docker-Compose / Kubernetes)
Relationships and Data Flows
• Processor Service:
→ Kafka Consumer from “raw-logs”
→ internal/processor logic → internal/elasticsearch client
→ Elasticsearch Index
→ Kafka Producer to “dlq” or “error-alerts” if needed
• API Service:
→ HTTP Client Requests → internal/api/router
→ Query Elasticsearch → Return results
→ Possibly produce alert messages to Kafka via internal/kafka/producer
• ES-Index Utility:
→ Reads configs → Calls ES APIs to set up index templates
Architectural Patterns and Design Principles
• Message-driven, event-streaming backbone (Kafka pub/sub)
• Worker pattern for parallel log processing
• Layered HTTP server (router → handlers → parsing/validation → service logic)
• Dependency injection via internal/setup & config module
• Single responsibility per binary (cmd/*)
Details
• Both normal and error paths (DLQ topics, error alerts)
• Bootstrap sequence: setup utility → create topics & indexes → start services
• Config management (configs/kafka vs. internal/config)
• If using Kubernetes, illustrate Deployments, Services, ConfigMaps, and how pods connect to Kafka and ES