How victoriametrics/victorialogs works
Project type and purpose
• VictoriaLogs is an open-source, high-performance distributed log database and accompanying toolset (agents, ingestion CLIs, selector CLIs, web UI, storage engine, deployment manifests).
• Its core is a log-storage engine (“victoria-logs” server) backed by the lib/logstorage library, plus multiple first-party clients (vlinsert, vlagent) and frontends (vlselect CLI, vlogscli, vmui web UI).
File structure
• Top-level “app” directory contains separate Go programs:
• victoria-logs (the storage/query server, can run single-node or clustered)
• vlagent (lightweight agent forwarding logs via remote_write)
• vlinsert (generic log ingestion CLI for various protocols: Elasticsearch, Loki, Datadog, syslog, journald, etc.)
• vlselect (CLI for querying logs via LogSQL over HTTP)
• vlogscli (generic CLI for log commands & pretty-printing)
• vlogsgenerator (log data generator for benchmarks)
• vlstorage (auxiliary storage service for cluster?)
• vmui (React + Vite web UI for log exploration)
• “lib/logstorage” is the core storage library used by the server(s).
• “deployment” & “dashboards” directories contain Docker Compose, Kubernetes/Helm charts, Grafana provisioning, etc.
• “docs” holds end-user documentation (architecture, data ingestion protocols, query language).
• CI/CD, GitHub workflows, Makefiles, Go modules are present for build/test/release pipelines.
README for additional insights
• README declares VictoriaLogs as “database for logs,” supports single-node and cluster modes.
• References to built-in playground and Grafana plugin hint at HTTP/gRPC APIs for query and ingestion.
• Deployment types (single vs cluster) imply a coordination layer (e.g., vmalert, vmstorage, vminsert in cluster).
Main components and their interactions
a. Clients & Agents (data producers)
• vlinsert (Go CLI): reads logs, transforms/formats, calls HTTP remote‐write API on laureate server.
• vlagent: tail-oriented agent, implements Prometheus remote_write for logs.
• filebeat/Fluentbit/Fluentd/Promtail/Telegraf/Vector adapters (packaged under deployment/docker/victorialogs) can also send logs.
b. Ingestion & Storage (data plane)
• victoria-logs server process:
• HTTP ingestion endpoints (Prometheus remote_write protocol, custom HTTP JSON).
• Query endpoints (LogSQL over HTTP, possibly gRPC for cluster).
• Persists incoming log rows via lib/logstorage: chunking, indexing, compression, retention.
• In cluster mode, multiple victoria-logs nodes communicate (sharding, replication).
c. Query & UI (data consumers)
• vlselect CLI: queries logs via HTTP using LogSQL, outputs text/JSON.
• vlogscli: generic CLI for fetching and pretty-printing logs.
• vmui: React SPA served via static assets + Go “web” service:
• Fetches data from server APIs (logs, facets, counts).
• Manages contexts, routing, state (React contexts, reducers).
d. Auxiliary & Deployment
• dashboards & Grafana plugin demonstrate external integration.
• Kubernetes/Helm and Docker Compose stacks tie together victoria-logs, Grafana, Alertmanager, Prometheus, etc.
• “vlogsgenerator” and “apptest” provide test harnesses/benchmarks.
Architectural patterns and design principles
• Microservices/monorepo: each “app/*” directory is a standalone service/CLI.
• Layered backend:
• Ingestion layer (remote_write, HTTP API)
• Storage library (lib/logstorage) with rich filter/query pipelines.
• Query layer (LogSQL, HTTP/gRPC).
• Frontend/back-end separation: React SPA (vmui) vs Go server API.
• Extensibility: adapters for multiple protocols under app/vlinsert, deployment/docker/victorialogs/*.
• Scalability: single-node vs cluster mode with sharding/replication (multiple server instances, vmstorage, vminsert?).
Final advice
• Every first-party app as its own component (monorepo’s “app/*” entries).
• How deployment (Docker/K8s) stitches them together with Grafana/Prometheus.
• Don’t overthink: break the system into ingestion, storage, query, UI, deployment layers, then map each directory/app to those layers.
• Keep it detailed but focused on main flows; minor helper libraries (prefixfilter) can be omitted or collapsed under the storage engine.