How keisku/gmon works
Project Type and Purpose
• gmon is a specialized monitoring tool for Go programs that leverages eBPF to track the creation and destruction of goroutines.
• Its purpose is to capture runtime goroutine events, log them with stack traces, and expose performance metrics (in OpenMetrics format) via an HTTP endpoint.
Analyzing the File Structure
• Top-Level Files:
– main.go: The main entry point for the application.
– gmon.sh: A shell script to build, install, and test the binary.
– go.mod and go.sum: Manage dependencies.
• ebpf Directory:
– Contains both Go code (e.g., config.go, event_handler.go, reporter.go) and embedded C code (in the “c” subdirectory with gmon.c, maps.h, etc.). This indicates that the system integrates kernel-level eBPF functionality with user-level Go code.
• bininfo Directory:
– Contains a dedicated module (bininfo.go) likely responsible for parsing binary or debugging information needed for monitoring (e.g., interacting with Go binaries for goroutine ID extraction).
• Others:
– Fixture directory: Provides a minimal Go module (go.mod, main.go) likely used as a test or sample target to validate gmon’s functionality.
– e2e_test.go: Contains end-to-end tests, which underline the importance of interaction between the monitoring tool and the target executables.
Reading the README
• The README provides usage instructions (command-line flags such as -path, -pid, -metrics, etc.), prerequisites (Linux Kernel 5.8+, amd64), and a demo of parsed logging output and exposed OpenMetrics metrics.
• It emphasizes both the runtime monitoring side (logging goroutine events) and observability via the metrics endpoint.
a. Main Components to Include:
• User-Provided Target Executable:
• The program that will be monitored by gmon.
• gmon Main Application (main.go):
• The orchestration layer that processes command-line arguments, initializes other components, loads eBPF programs, and starts monitoring.
• eBPF Engine (ebpf Package):
• Comprised of Go code (config.go, event_handler.go) and native C code (in the “c” subfolder). This module loads and interacts with the BPF ring buffer to capture kernel-level events about goroutine creation and destruction.
• Bininfo Module:
• Responsible for retrieving metadata or offsets from the target binary to assist the eBPF code in correctly identifying goroutine information.
• Reporter/Logging & Metrics Module:
• As seen in reporter.go and reporter_test.go, this component logs event details (e.g., goroutine creation with call stacks) and exposes these details as metrics in the OpenMetrics format.
• HTTP Server:
• While not explicitly a separate package, the application exposes a /metrics endpoint (as specified by the -metrics flag) and optionally a pprof endpoint for profiling.
b. Relationships and Interactions:
• The main application loads the eBPF programs using the ebpf module.
• The eBPF engine hooks into the running target process’s kernel events (using a BPF ring buffer mechanism) to monitor goroutine lifecycle events.
• When an event occurs (e.g., goroutine creation or exit), the event_handler processes it and passes it to the Reporter module.
• The Reporter module then logs the events to stdout (with detailed stack trace information) and updates the metrics counters/histograms available via the HTTP metrics endpoint.
• The bininfo component ties into this flow by providing necessary binary metadata that allows the eBPF engine to correctly interpret the runtime memory or offset details in the target binary.
c. Architectural Patterns and Design Principles:
• Separation of Concerns:
• Each module (eBPF loading, binary info extraction, event reporting) serves a specific role, making the design modular.
• Layered Architecture:
• The lowest level (kernel/eBPF interaction) is abstracted through a Go package, ensuring that the rest of the application remains agnostic of low-level kernel details.
• Extensibility:
• With clear separations (monitoring, logging, metrics), the design supports adding more event types or further integrations (e.g., additional metrics collectors or output methods).
d. Relevant Technologies/Frameworks:
• Go (of course, as the main programming language)
• eBPF and BPF ring buffer (for kernel-level event capture)
• C for low-level eBPF programs and map definitions
• OpenMetrics (for exposing performance metrics)
• Standard libraries for HTTP handling (metrics endpoint and pprof)