# victoriametrics/victorialogs architecture

> Architecture diagram of the GitHub repository victoriametrics/victorialogs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/victoriametrics/victorialogs
- Repository: https://github.com/victoriametrics/victorialogs

## Overview

##### Project type and purpose
- VictoriaLogs is an open-source, high-performance distributed log database and accompanying toolset (agents, ingestion CLIs, selector CLIs, web UI, storage engine, deployment manifests).
- Its core is a log-storage engine (“victoria-logs” server) backed by the lib/logstorage library, plus multiple first-party clients (vlinsert, vlagent) and frontends (vlselect CLI, vlogscli, vmui web UI).

##### File structure
- Top-level “app” directory contains separate Go programs:
- victoria-logs (the storage/query server, can run single-node or clustered)
- vlagent (lightweight agent forwarding logs via remote_write)
- vlinsert (generic log ingestion CLI for various protocols: Elasticsearch, Loki, Datadog, syslog, journald, etc.)
- vlselect (CLI for querying logs via LogSQL over HTTP)
- vlogscli (generic CLI for log commands & pretty-printing)
- vlogsgenerator (log data generator for benchmarks)
- vlstorage (auxiliary storage service for cluster?)
- vmui (React + Vite web UI for log exploration)
- “lib/logstorage” is the core storage library used by the server(s).
- “deployment” & “dashboards” directories contain Docker Compose, Kubernetes/Helm charts, Grafana provisioning, etc.
- “docs” holds end-user documentation (architecture, data ingestion protocols, query language).
- CI/CD, GitHub workflows, Makefiles, Go modules are present for build/test/release pipelines.

##### README for additional insights
- README declares VictoriaLogs as “database for logs,” supports single-node and cluster modes.
- References to built-in playground and Grafana plugin hint at HTTP/gRPC APIs for query and ingestion.
- Deployment types (single vs cluster) imply a coordination layer (e.g., vmalert, vmstorage, vminsert in cluster).

##### Main components and their interactions
a. Clients & Agents (data producers)
- vlinsert (Go CLI): reads logs, transforms/formats, calls HTTP remote‐write API on laureate server.
- vlagent: tail-oriented agent, implements Prometheus remote_write for logs.
- filebeat/Fluentbit/Fluentd/Promtail/Telegraf/Vector adapters (packaged under deployment/docker/victorialogs) can also send logs.
b. Ingestion & Storage (data plane)
- victoria-logs server process:
- HTTP ingestion endpoints (Prometheus remote_write protocol, custom HTTP JSON).
- Query endpoints (LogSQL over HTTP, possibly gRPC for cluster).
- Persists incoming log rows via lib/logstorage: chunking, indexing, compression, retention.
- In cluster mode, multiple victoria-logs nodes communicate (sharding, replication).
c. Query & UI (data consumers)
- vlselect CLI: queries logs via HTTP using LogSQL, outputs text/JSON.
- vlogscli: generic CLI for fetching and pretty-printing logs.
- vmui: React SPA served via static assets + Go “web” service:
- Fetches data from server APIs (logs, facets, counts).
- Manages contexts, routing, state (React contexts, reducers).
d. Auxiliary & Deployment
- dashboards & Grafana plugin demonstrate external integration.
- Kubernetes/Helm and Docker Compose stacks tie together victoria-logs, Grafana, Alertmanager, Prometheus, etc.
- “vlogsgenerator” and “apptest” provide test harnesses/benchmarks.

##### Architectural patterns and design principles
- Microservices/monorepo: each “app/*” directory is a standalone service/CLI.
- Layered backend:
- Ingestion layer (remote_write, HTTP API)
- Storage library (lib/logstorage) with rich filter/query pipelines.
- Query layer (LogSQL, HTTP/gRPC).
- Frontend/back-end separation: React SPA (vmui) vs Go server API.
- Extensibility: adapters for multiple protocols under app/vlinsert, deployment/docker/victorialogs/*.
- Scalability: single-node vs cluster mode with sharding/replication (multiple server instances, vmstorage, vminsert?).

##### Final advice
- Every first-party app as its own component (monorepo’s “app/*” entries).
- How deployment (Docker/K8s) stitches them together with Grafana/Prometheus.
- Don’t overthink: break the system into ingestion, storage, query, UI, deployment layers, then map each directory/app to those layers.
- Keep it detailed but focused on main flows; minor helper libraries (prefixfilter) can be omitted or collapsed under the storage engine.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Producers"
        DP_vlinsert["vlinsert CLI"]:::cli
        DP_vlagent["vlagent Agent"]:::cli
    end

    subgraph "Ingestion & Query API"
        VLServer["victoria-logs Server\n(HTTP Ingestion & Query)"]:::service
    end

    subgraph "Storage Layer"
        Storage["Core Storage Engine\n(lib/logstorage)"]:::storage
    end

    subgraph "Consumers"
        VLSel["vlselect CLI"]:::cli
        VLogsCLI["vlogscli CLI"]:::cli
        VMUI_BE["vmui Go Backend\n(serves UI assets)"]:::service
        VMUI_UI["vmui Web UI\n(React SPA)"]:::ui
        Grafana["Grafana Dashboards"]:::external
    end

    subgraph "Deployment Orchestrator"
        DockerCompose["Docker Compose\n(single-node)"]:::deployment
        Helm["K8s Helm Charts"]:::deployment
    end

    %% Data flows
    DP_vlinsert -->|"HTTP remote_write"| VLServer
    DP_vlagent -->|"Prometheus remote_write"| VLServer

    VLServer -->|"Persist logs"| Storage
    Storage -->|"Query data"| VLServer

    VLServer -->|"LogSQL over HTTP"| VLSel
    VLServer -->|"LogSQL over HTTP"| VLogsCLI
    VLServer -->|"HTTP API"| VMUI_BE
    VMUI_BE -->|"Serves SPA"| VMUI_UI

    %% Deployment connections
    DockerCompose --> VLServer
    DockerCompose --> VMUI_BE
    DockerCompose --> Grafana
    Helm --> VLServer
    Helm --> VMUI_BE
    Helm --> Grafana

    %% Click events
    click DP_vlinsert "https://github.com/victoriametrics/victorialogs/blob/master/app/vlinsert/main.go"
    click DP_vlagent "https://github.com/victoriametrics/victorialogs/blob/master/app/vlagent/main.go"
    click VLServer "https://github.com/victoriametrics/victorialogs/blob/master/app/victoria-logs/main.go"
    click Storage "https://github.com/victoriametrics/victorialogs/tree/master/lib/logstorage/"
    click VLSel "https://github.com/victoriametrics/victorialogs/blob/master/app/vlselect/main.go"
    click VLogsCLI "https://github.com/victoriametrics/victorialogs/blob/master/app/vlogscli/main.go"
    click VMUI_UI "https://github.com/victoriametrics/victorialogs/blob/master/app/vmui/packages/vmui/src/index.tsx"
    click VMUI_BE "https://github.com/victoriametrics/victorialogs/blob/master/app/vmui/packages/vmui/nginx/nginx.conf"
    click Grafana "https://github.com/victoriametrics/victorialogs/blob/master/deployment/docker/provisioning/dashboards/dashboard.yml"
    click DockerCompose "https://github.com/victoriametrics/victorialogs/blob/master/deployment/docker/compose-vl-single.yml"
    click Helm "https://github.com/victoriametrics/victorialogs/tree/master/deployment/k8s/helm/"

    %% Styles
    classDef service fill:#1f77b4,stroke:#333,stroke-width:1px,color:#fff;
    classDef cli fill:#7f7f7f,stroke:#333,stroke-width:1px,color:#fff;
    classDef storage fill:#2ca02c,stroke:#333,stroke-width:1px,color:#fff,shape:cylinder;
    classDef ui fill:#ff7f0e,stroke:#333,stroke-width:1px,color:#fff;
    classDef deployment fill:#8c564b,stroke:#333,stroke-width:1px,color:#fff;
    classDef external fill:#d62728,stroke:#333,stroke-width:1px,color:#fff;
```
