# edgio/vflow architecture

> Architecture diagram of the GitHub repository edgio/vflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/edgio/vflow
- Repository: https://github.com/edgio/vflow

## Overview

1. **Overall purpose**
   - This repository is a network flow collector/forwarder for **IPFIX, sFlow, NetFlow v5, and NetFlow v9**.
   - The core runtime is the **`vflow` Go service**, which receives exporter traffic, decodes it, optionally mirrors it, and can publish normalized flow data to downstream systems.

2. **Ingress layer**
   - The collector listens on the standard flow-protocol network entry points, with OS-specific socket handling in `vflow/ipfix_*.go` and `vflow/sflow_*.go`.
   - The read path is separated into a generic **`reader`** package and protocol-specific ingest code in **`vflow`**, so packet acquisition is isolated from protocol parsing.

3. **Protocol decoding boundary**
   - Decoding is split by protocol family:
     - **`ipfix/`** for RFC7011/IPFIX
     - **`netflow/v5/`** and **`netflow/v9/`**
     - **`sflow/`**
   - These packages turn wire format into structured flow records, using protocol-specific templates, field definitions, and marshalling logic.

4. **Template and schema handling**
   - IPFIX and NetFlow v9 both rely on **template-based decoding**, so there are cache/memory helpers such as `ipfix/memcache.go` and `netflow/v9/memcache.go`.
   - The IPFIX package also includes an RFC5102 model mapping (`rfc5102_model.go`), indicating normalization against known element IDs and field semantics.

5. **Packet parsing subsystem**
   - The **`packet/`** package handles low-level network header parsing for **Ethernet, IPv4/IPv6, transport, and ICMP**.
   - This is used especially by sFlow raw-header decoding, where the collector needs to reconstruct L2/L3/L4 context from sampled packets.

6. **Main processing/runtime boundary**
   - The **`vflow/`** package is the orchestration layer:
     - config/options
     - protocol-specific ingest entry points
     - stats
     - platform-specific implementations
   - This is the boundary that wires listeners, decoders, and downstream outputs together.

7. **Egress and fan-out**
   - The **`producer/`** package abstracts publishing to external messaging/streaming systems:
     - **Kafka** (`sarama.go`, `segmentio.go`)
     - **NSQ**
     - **NATS**
     - raw socket forwarding
   - This makes the architecture multi-sink: decoded flows can be sent to one or more external pipelines without changing the ingest path.

8. **Mirroring / replication path**
   - The **`mirror/`** package is a separate forwarding path for **replicating IPFIX and sFlow** to third-party collectors.
   - It includes IPv4/IPv6 and UDP helpers, which suggests it operates at the datagram level rather than as a decoded-flow sink.

9. **Monitoring and observability**
   - The **`monitor/`** subtree exposes collector health and metrics.
   - It supports a store abstraction (`monitor/store/`) with implementations such as **InfluxDB** and a generic TSDB interface.
   - The README also references **Prometheus** and REST-style monitoring, so the service has an internal metrics plane separate from flow data.

10. **Operational and deployment packaging**
    - The repo includes first-class deployment artifacts: **Dockerfile**, **docker-compose**, **Kubernetes manifest**, and Linux packaging scripts for **deb/rpm/service units**.
    - This indicates the collector is intended to run as a long-lived network service, not just a library.

11. **Consumer examples and downstream analytics**
    - The **`consumers/`** directory contains example downstream processors for **CLI**, **ClickHouse**, **MemSQL**, and **Spark**.
    - These are not part of the collector’s core path; they show how decoded flow output can be consumed by analytics/warehouse systems.

12. **Load generation and testing**
    - The **`stress/`** and `stress/hammer` packages provide synthetic IPFIX/sFlow traffic generation for performance testing.
    - This is important architecturally because throughput and packet-processing behavior are central to the design.

13. **Typical end-to-end data flow**
    - Exporter device → UDP/raw socket listener → protocol decoder/template cache → normalized flow record → optional mirror/producer → external queue/collector/analytics store → monitoring subsystem tracks health and volume.

14. **Key architectural boundaries**
    - The repository is organized around three clear separations:
      - **wire ingestion**
      - **protocol normalization**
      - **external distribution/observability**
    - That separation is what makes the system scalable across multiple flow protocols and multiple output backends.

Main entry points: [`vflow/vflow.go`](https://github.com/edgio/vflow/blob/master/vflow/vflow.go) (vflow), [`reader/reader.go`](https://github.com/edgio/vflow/blob/master/reader/reader.go) (Reader), [`vflow/ipfix.go`](https://github.com/edgio/vflow/blob/master/vflow/ipfix.go) (IPFIX ingest), [`vflow/sflow.go`](https://github.com/edgio/vflow/blob/master/vflow/sflow.go) (sFlow ingest)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingress"]
  node_reader["Reader<br/>acquire<br/>[reader.go]"]
  node_ipfix_ingest["IPFIX ingest<br/>listener<br/>[ipfix.go]"]
  node_sflow_ingest["sFlow ingest<br/>listener<br/>[sflow.go]"]
  node_nfv5_ingest["NetFlow v5<br/>listener<br/>[netflow_v5.go]"]
  node_nfv9_ingest["NetFlow v9<br/>listener<br/>[netflow_v9.go]"]
end

subgraph group_decode["Decoding"]
  node_packet["Packet<br/>parsers<br/>[packet.go]"]
  node_ipfix_decode["IPFIX decode<br/>protocol parser<br/>[decoder.go]"]
  node_ipfix_cache["IPFIX cache<br/>template cache<br/>[memcache.go]"]
  node_nfv5_decode["v5 decode<br/>protocol parser<br/>[decoder.go]"]
  node_nfv9_decode["v9 decode<br/>protocol parser<br/>[decoder.go]"]
  node_nfv9_cache["v9 cache<br/>template cache<br/>[memcache.go]"]
  node_sflow_decode["sFlow decode<br/>protocol parser<br/>[decoder.go]"]
end

subgraph group_runtime["Runtime"]
  node_vflow_runtime["vflow<br/>Go service<br/>[vflow.go]"]
  node_vflow_options["Options<br/>config<br/>[options.go]"]
  node_vflow_stats["Stats<br/>metrics<br/>[stats.go]"]
end

subgraph group_egress["Egress"]
  node_mirror["Mirror<br/>replicator<br/>[mirror.go]"]
  node_producer["Producer<br/>fan-out<br/>[producer.go]"]
  node_monitor["Monitor<br/>observability<br/>[monitor.go]"]
end

subgraph group_ops["Ops"]
  node_consumers["Consumers<br/>analytics examples<br/>[main.go]"]
  node_stress["Stress<br/>load generator<br/>[stress.go]"]
  node_deploy["Deploy<br/>packaging"]
end

node_vflow_runtime -->|"loads"| node_vflow_options
node_vflow_runtime -->|"reports"| node_vflow_stats
node_vflow_runtime -->|"reads"| node_reader
node_reader -->|"feeds"| node_ipfix_ingest
node_reader -->|"feeds"| node_sflow_ingest
node_reader -->|"feeds"| node_nfv5_ingest
node_reader -->|"feeds"| node_nfv9_ingest
node_ipfix_ingest -->|"decodes"| node_ipfix_decode
node_ipfix_decode -->|"uses"| node_ipfix_cache
node_nfv9_ingest -->|"decodes"| node_nfv9_decode
node_nfv9_decode -->|"uses"| node_nfv9_cache
node_nfv5_ingest -->|"decodes"| node_nfv5_decode
node_sflow_ingest -->|"decodes"| node_sflow_decode
node_sflow_decode -->|"parses headers"| node_packet
node_ipfix_ingest -->|"can mirror"| node_mirror
node_sflow_ingest -->|"can mirror"| node_mirror
node_vflow_runtime -->|"publishes"| node_producer
node_vflow_runtime -->|"exposes"| node_monitor
node_producer -->|"feeds"| node_consumers
node_stress -->|"drives"| node_reader
node_deploy -->|"packages"| node_vflow_runtime

click node_vflow_runtime "https://github.com/edgio/vflow/blob/master/vflow/vflow.go"
click node_vflow_options "https://github.com/edgio/vflow/blob/master/vflow/options.go"
click node_vflow_stats "https://github.com/edgio/vflow/blob/master/vflow/stats.go"
click node_reader "https://github.com/edgio/vflow/blob/master/reader/reader.go"
click node_ipfix_ingest "https://github.com/edgio/vflow/blob/master/vflow/ipfix.go"
click node_sflow_ingest "https://github.com/edgio/vflow/blob/master/vflow/sflow.go"
click node_nfv5_ingest "https://github.com/edgio/vflow/blob/master/vflow/netflow_v5.go"
click node_nfv9_ingest "https://github.com/edgio/vflow/blob/master/vflow/netflow_v9.go"
click node_packet "https://github.com/edgio/vflow/blob/master/packet/packet.go"
click node_ipfix_decode "https://github.com/edgio/vflow/blob/master/ipfix/decoder.go"
click node_ipfix_cache "https://github.com/edgio/vflow/blob/master/ipfix/memcache.go"
click node_nfv5_decode "https://github.com/edgio/vflow/blob/master/netflow/v5/decoder.go"
click node_nfv9_decode "https://github.com/edgio/vflow/blob/master/netflow/v9/decoder.go"
click node_nfv9_cache "https://github.com/edgio/vflow/blob/master/netflow/v9/memcache.go"
click node_sflow_decode "https://github.com/edgio/vflow/blob/master/sflow/decoder.go"
click node_mirror "https://github.com/edgio/vflow/blob/master/mirror/mirror.go"
click node_producer "https://github.com/edgio/vflow/blob/master/producer/producer.go"
click node_monitor "https://github.com/edgio/vflow/blob/master/monitor/monitor.go"
click node_consumers "https://github.com/edgio/vflow/blob/master/consumers/cli/main.go"
click node_stress "https://github.com/edgio/vflow/blob/master/stress/stress.go"
click node_deploy "https://github.com/edgio/vflow/tree/master/Dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_reader,node_ipfix_ingest,node_sflow_ingest,node_nfv5_ingest,node_nfv9_ingest toneBlue
class node_packet,node_ipfix_decode,node_ipfix_cache,node_nfv5_decode,node_nfv9_decode,node_nfv9_cache,node_sflow_decode toneAmber
class node_vflow_runtime,node_vflow_options,node_vflow_stats toneMint
class node_mirror,node_producer,node_monitor toneRose
class node_consumers,node_stress,node_deploy toneIndigo
```

## Components

### Ingress

Wire listeners and packet acquisition

- [Reader](https://github.com/edgio/vflow/blob/master/reader/reader.go): acquire · Shared read path for incoming flow datagrams · `reader/reader.go`
- [IPFIX ingest](https://github.com/edgio/vflow/blob/master/vflow/ipfix.go): listener · IPFIX socket handling and ingest entry points · `vflow/ipfix.go`
- [sFlow ingest](https://github.com/edgio/vflow/blob/master/vflow/sflow.go): listener · sFlow socket handling and ingest entry points · `vflow/sflow.go`
- [NetFlow v5](https://github.com/edgio/vflow/blob/master/vflow/netflow_v5.go): listener · NetFlow v5 ingest entry point · `vflow/netflow_v5.go`
- [NetFlow v9](https://github.com/edgio/vflow/blob/master/vflow/netflow_v9.go): listener · NetFlow v9 ingest entry point · `vflow/netflow_v9.go`

### Decoding

Protocol-specific normalization and templates

- [Packet](https://github.com/edgio/vflow/blob/master/packet/packet.go): parsers · Ethernet, IP, transport, and ICMP header parsing · `packet/packet.go`
- [IPFIX decode](https://github.com/edgio/vflow/blob/master/ipfix/decoder.go): protocol parser · IPFIX decoding, interpretation, and marshal logic · `ipfix/decoder.go`
- [IPFIX cache](https://github.com/edgio/vflow/blob/master/ipfix/memcache.go): template cache · Template and memory cache support for IPFIX · `ipfix/memcache.go`
- [v5 decode](https://github.com/edgio/vflow/blob/master/netflow/v5/decoder.go): protocol parser · NetFlow v5 record decoding and marshalling · `netflow/v5/decoder.go`
- [v9 decode](https://github.com/edgio/vflow/blob/master/netflow/v9/decoder.go): protocol parser · NetFlow v9 template-based decoding and marshalling · `netflow/v9/decoder.go`
- [v9 cache](https://github.com/edgio/vflow/blob/master/netflow/v9/memcache.go): template cache · Template cache support for NetFlow v9 · `netflow/v9/memcache.go`
- [sFlow decode](https://github.com/edgio/vflow/blob/master/sflow/decoder.go): protocol parser · sFlow flow and counter sample decoding · `sflow/decoder.go`

### Runtime

Collector orchestration and stats

- [vflow](https://github.com/edgio/vflow/blob/master/vflow/vflow.go): Go service · Main collector runtime that wires listeners, decoders, outputs, and stats · `vflow/vflow.go`
- [Options](https://github.com/edgio/vflow/blob/master/vflow/options.go): config · Collector configuration and runtime options · `vflow/options.go`
- [Stats](https://github.com/edgio/vflow/blob/master/vflow/stats.go): metrics · Internal counters and runtime health tracking · `vflow/stats.go`

### Egress

Mirroring, producers, and monitoring

- [Mirror](https://github.com/edgio/vflow/blob/master/mirror/mirror.go): replicator · Datagram-level replication for IPFIX and sFlow · `mirror/mirror.go`
- [Producer](https://github.com/edgio/vflow/blob/master/producer/producer.go): fan-out · Abstraction for Kafka, NSQ, NATS, and raw socket outputs · `producer/producer.go`
- [Monitor](https://github.com/edgio/vflow/blob/master/monitor/monitor.go): observability · Collector monitoring and metrics exposure · `monitor/monitor.go`

### Ops

Deployment and example consumers

- [Consumers](https://github.com/edgio/vflow/blob/master/consumers/cli/main.go): analytics examples · Example downstream processors for CLI, ClickHouse, MemSQL, and Spark · `consumers/cli/main.go`
- [Stress](https://github.com/edgio/vflow/blob/master/stress/stress.go): load generator · Synthetic traffic generation for throughput testing · `stress/stress.go`
- [Deploy](https://github.com/edgio/vflow/tree/master/Dockerfile): packaging · Docker, Kubernetes, and service packaging assets · `Dockerfile`

## Connections

- vflow → Options: loads · Runtime reads configuration
- vflow → Stats: reports · Runtime updates internal metrics
- vflow → Reader: reads · Collector acquires incoming datagrams
- Reader → IPFIX ingest: feeds · Ingress path for IPFIX traffic
- Reader → sFlow ingest: feeds · Ingress path for sFlow traffic
- Reader → NetFlow v5: feeds · Ingress path for NetFlow v5 traffic
- Reader → NetFlow v9: feeds · Ingress path for NetFlow v9 traffic
- IPFIX ingest → IPFIX decode: decodes · IPFIX payloads are normalized into structured records
- IPFIX decode → IPFIX cache: uses · Template-based decoding relies on cached state
- NetFlow v9 → v9 decode: decodes · NetFlow v9 payloads are normalized into structured records
- v9 decode → v9 cache: uses · Template-based decoding relies on cached state
- NetFlow v5 → v5 decode: decodes · NetFlow v5 is decoded directly without templates
- sFlow ingest → sFlow decode: decodes · sFlow samples are parsed into flow records
- sFlow decode → Packet: parses headers · Raw packet samples use low-level header parsing
- IPFIX ingest → Mirror: can mirror · IPFIX datagrams may be replicated upstream
- sFlow ingest → Mirror: can mirror · sFlow datagrams may be replicated upstream
- vflow → Producer: publishes · Decoded flows can be sent to external sinks
- vflow → Monitor: exposes · Runtime health and metrics are exported separately
- Producer → Consumers: feeds · Example downstream consumers show analytics integrations
- Stress → Reader: drives · Load generation exercises the ingest path
- Deploy → vflow: packages · Deployment artifacts ship the collector service
