How netsampler/goflow2 works
Project type and purpose
• GoFlow2 is a high-throughput, horizontally scalable network flow collector written in Go.
• It listens for NetFlow/IPFIX and sFlow packets, decodes them to a common internal format, optionally enriches them, and then emits them via different output transports (file, stdout, Kafka) in either JSON, text or Protobuf.
• It is not a full “frontend-backend” web app; it is a backend data-pipeline service/tool.
File structure
Top-level directories reveal a modular, pipeline-style architecture:
• cmd/ – two entry-points:
– goflow2 collector (main.go, mapping.yaml)
– enricher (main.go, flowext.proto, flowext.pb.go)
• decoders/ – protocol‐specific decoding libraries (netflow, netflowlegacy, sflow), plus shared utils
• producer/ – transforms decoded samples into outbound records (raw vs. Protobuf)
• format/ – marshaling strategies (binary, JSON, text)
• transport/ – output adapters (kafka, file)
• metrics/ – collects and exposes Prometheus metrics at each pipeline stage
• utils/ – common utilities (debugging, UDP handling, templates)
• pb/ – canonical Protobuf schema for “flow” messages
• compose/ – example Docker Compose pipelines (ELK and Kafka+ClickHouse+Grafana)
Configuration & CI files (Makefile, Dockerfile, GitHub workflows) support building, testing, containerization and releases.
Key architectural insights from README
• The pipeline is structured as a chain of pluggable components (decoder → producer → formatter → transport).
• Enrichment is a separate CLI tool that reads Protobuf from stdin and writes enriched Protobuf to stdout.
• External integrations shown in compose/: Kafka, ClickHouse, Grafana, ELK, Logstash.
• Metrics are pushed to Prometheus.
Main components
A. Input Layer
– UDP listeners in cmd/goflow2 (two protocols: sFlow on UDP 6343, NetFlow/IPFIX on UDP 2055)
B. Decoder Layer
– netflow decoder (v5, v9/IPFIX with template cache)
– sflow decoder (v5)
C. Producer Layer
– raw producer (pass records as Go structs)
– Protobuf producer (uses pb/flow.proto)
D. Formatter Layer
– JSON, text, or binary (via format/json, format/text, format/binary)
– Custom field mapping (cmd/goflow2/mapping.yaml)
E. Transport Layer
– File or stdout (transport/file)
– Kafka (transport/kafka using Sarama client)
F. Enricher (optional)
– cmd/enricher reads Protobuf, enriches via MaxMind DBs, outputs Protobuf
G. Metrics & Monitoring
– metrics exposed on HTTP (Prometheus endpoint, metrics/…)
H. External Services
– Kafka cluster
– File system or log aggregator (e.g. Loki, Splunk)
– Downstream storage/graphing: ClickHouse + Grafana or Elasticsearch + Kibana
Relationships and data flow
UDP Listeners → Decoder → Producer → Formatter → Transport → External Sink
• At each stage—Decoder, Producer, Transport—metrics are emitted to Prometheus.
• Enricher is a side-pipeline that consumes Protobuf output and feeds back into transport/formatter.
• Mapping component intercepts in Producer stage to apply custom field mappings.
Architectural patterns & design principles
• Pipeline/chain of responsibility: each stage implements a shared interface.
• Strategy pattern: formatters and transports are pluggable at runtime via CLI flags.
• Separation of concerns: decoding, producing, formatting, transport and enrichment are isolated packages.
• Modular, star-topology: central CLI orchestrates independent modules.
Essential details
• Ports (6343/2055 for UDP).
• Configurable mapping file for IPFIX templates.
• CLI flags that select strategies (-format, ‑transport).
• Protobuf schema version (pb/flow.proto).