How edgio/vflow works
1. Overall purpose
- This repository is a network flow collector/forwarder for IPFIX, sFlow, NetFlow v5, and NetFlow v9.
- The core runtime is the vflow Go service, which receives exporter traffic, decodes it, optionally mirrors it, and can publish normalized flow data to downstream systems.
2. Ingress layer
- The collector listens on the standard flow-protocol network entry points, with OS-specific socket handling in vflow/ipfix_*.go and vflow/sflow_*.go.
- The read path is separated into a generic reader package and protocol-specific ingest code in vflow, so packet acquisition is isolated from protocol parsing.
3. Protocol decoding boundary
- Decoding is split by protocol family:
- ipfix/ for RFC7011/IPFIX
- netflow/v5/ and netflow/v9/
- sflow/
- These packages turn wire format into structured flow records, using protocol-specific templates, field definitions, and marshalling logic.
4. Template and schema handling
- IPFIX and NetFlow v9 both rely on template-based decoding, so there are cache/memory helpers such as ipfix/memcache.go and netflow/v9/memcache.go.
- The IPFIX package also includes an RFC5102 model mapping (rfc5102_model.go), indicating normalization against known element IDs and field semantics.
5. Packet parsing subsystem
- The packet/ package handles low-level network header parsing for Ethernet, IPv4/IPv6, transport, and ICMP.
- This is used especially by sFlow raw-header decoding, where the collector needs to reconstruct L2/L3/L4 context from sampled packets.
6. Main processing/runtime boundary
- The vflow/ package is the orchestration layer:
- config/options
- protocol-specific ingest entry points
- stats
- platform-specific implementations
- This is the boundary that wires listeners, decoders, and downstream outputs together.
7. Egress and fan-out
- The producer/ package abstracts publishing to external messaging/streaming systems:
- Kafka (sarama.go, segmentio.go)
- NSQ
- NATS
- raw socket forwarding
- This makes the architecture multi-sink: decoded flows can be sent to one or more external pipelines without changing the ingest path.
8. Mirroring / replication path
- The mirror/ package is a separate forwarding path for replicating IPFIX and sFlow to third-party collectors.
- It includes IPv4/IPv6 and UDP helpers, which suggests it operates at the datagram level rather than as a decoded-flow sink.
9. Monitoring and observability
- The monitor/ subtree exposes collector health and metrics.
- It supports a store abstraction (monitor/store/) with implementations such as InfluxDB and a generic TSDB interface.
- The README also references Prometheus and REST-style monitoring, so the service has an internal metrics plane separate from flow data.
10. Operational and deployment packaging
- The repo includes first-class deployment artifacts: Dockerfile, docker-compose, Kubernetes manifest, and Linux packaging scripts for deb/rpm/service units.
- This indicates the collector is intended to run as a long-lived network service, not just a library.
11. Consumer examples and downstream analytics
- The consumers/ directory contains example downstream processors for CLI, ClickHouse, MemSQL, and Spark.
- These are not part of the collector’s core path; they show how decoded flow output can be consumed by analytics/warehouse systems.
12. Load generation and testing
- The stress/ and stress/hammer packages provide synthetic IPFIX/sFlow traffic generation for performance testing.
- This is important architecturally because throughput and packet-processing behavior are central to the design.
13. Typical end-to-end data flow
- Exporter device → UDP/raw socket listener → protocol decoder/template cache → normalized flow record → optional mirror/producer → external queue/collector/analytics store → monitoring subsystem tracks health and volume.
14. Key architectural boundaries
- The repository is organized around three clear separations:
- wire ingestion
- protocol normalization
- external distribution/observability
- That separation is what makes the system scalable across multiple flow protocols and multiple output backends.
Main entry points: vflow.go vflow · reader.go Reader · ipfix.go IPFIX ingest · sflow.go sFlow ingest
Components 21
Ingress
Wire listeners and packet acquisition
- Reader · Shared read path for incoming flow datagrams
reader/reader.go - IPFIX ingest · IPFIX socket handling and ingest entry points
vflow/ipfix.go - sFlow ingest · sFlow socket handling and ingest entry points
vflow/sflow.go - NetFlow v5 · NetFlow v5 ingest entry point
vflow/netflow_v5.go - NetFlow v9 · NetFlow v9 ingest entry point
vflow/netflow_v9.go
Decoding
Protocol-specific normalization and templates
- Packet · Ethernet, IP, transport, and ICMP header parsing
packet/packet.go - IPFIX decode · IPFIX decoding, interpretation, and marshal logic
ipfix/decoder.go - IPFIX cache · Template and memory cache support for IPFIX
ipfix/memcache.go - v5 decode · NetFlow v5 record decoding and marshalling
netflow/v5/decoder.go - v9 decode · NetFlow v9 template-based decoding and marshalling
netflow/v9/decoder.go - v9 cache · Template cache support for NetFlow v9
netflow/v9/memcache.go - sFlow decode · sFlow flow and counter sample decoding
sflow/decoder.go
Runtime
Collector orchestration and stats
Egress
Mirroring, producers, and monitoring
Connections 21
- vflow → Options: loads · Runtime reads configuration
- vflow → Stats: reports · Runtime updates internal metrics
- vflow → Reader: reads · Collector acquires incoming datagrams
- Reader → IPFIX ingest: feeds · Ingress path for IPFIX traffic
- Reader → sFlow ingest: feeds · Ingress path for sFlow traffic
- Reader → NetFlow v5: feeds · Ingress path for NetFlow v5 traffic
- Reader → NetFlow v9: feeds · Ingress path for NetFlow v9 traffic
- IPFIX ingest → IPFIX decode: decodes · IPFIX payloads are normalized into structured records
- IPFIX decode → IPFIX cache: uses · Template-based decoding relies on cached state
- NetFlow v9 → v9 decode: decodes · NetFlow v9 payloads are normalized into structured records
- v9 decode → v9 cache: uses · Template-based decoding relies on cached state
- NetFlow v5 → v5 decode: decodes · NetFlow v5 is decoded directly without templates
- sFlow ingest → sFlow decode: decodes · sFlow samples are parsed into flow records
- sFlow decode → Packet: parses headers · Raw packet samples use low-level header parsing
- IPFIX ingest → Mirror: can mirror · IPFIX datagrams may be replicated upstream
- sFlow ingest → Mirror: can mirror · sFlow datagrams may be replicated upstream
- vflow → Producer: publishes · Decoded flows can be sent to external sinks
- vflow → Monitor: exposes · Runtime health and metrics are exported separately
- Producer → Consumers: feeds · Example downstream consumers show analytics integrations
- Stress → Reader: drives · Load generation exercises the ingest path
- Deploy → vflow: packages · Deployment artifacts ship the collector service