# netsampler/goflow2 architecture

> Architecture diagram of the GitHub repository netsampler/goflow2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netsampler/goflow2
- Repository: https://github.com/netsampler/goflow2

## Overview

##### Project type and purpose
- GoFlow2 is a high-throughput, horizontally scalable network flow collector written in Go.
- It listens for NetFlow/IPFIX and sFlow packets, decodes them to a common internal format, optionally enriches them, and then emits them via different output transports (file, stdout, Kafka) in either JSON, text or Protobuf.
- It is not a full “frontend-backend” web app; it is a backend data-pipeline service/tool.

##### File structure
Top-level directories reveal a modular, pipeline-style architecture:
- cmd/ – two entry-points:
– goflow2 collector (main.go, mapping.yaml)
– enricher (main.go, flowext.proto, flowext.pb.go)
- decoders/ – protocol‐specific decoding libraries (netflow, netflowlegacy, sflow), plus shared utils
- producer/ – transforms decoded samples into outbound records (raw vs. Protobuf)
- format/ – marshaling strategies (binary, JSON, text)
- transport/ – output adapters (kafka, file)
- metrics/ – collects and exposes Prometheus metrics at each pipeline stage
- utils/ – common utilities (debugging, UDP handling, templates)
- pb/ – canonical Protobuf schema for “flow” messages
- compose/ – example Docker Compose pipelines (ELK and Kafka+ClickHouse+Grafana)

Configuration & CI files (Makefile, Dockerfile, GitHub workflows) support building, testing, containerization and releases.

##### Key architectural insights from README
- The pipeline is structured as a chain of pluggable components (decoder → producer → formatter → transport).
- Enrichment is a separate CLI tool that reads Protobuf from stdin and writes enriched Protobuf to stdout.
- External integrations shown in compose/: Kafka, ClickHouse, Grafana, ELK, Logstash.
- Metrics are pushed to Prometheus.

##### Main components
A. Input Layer
– UDP listeners in cmd/goflow2 (two protocols: sFlow on UDP 6343, NetFlow/IPFIX on UDP 2055)
B. Decoder Layer
– netflow decoder (v5, v9/IPFIX with template cache)
– sflow decoder (v5)
C. Producer Layer
– raw producer (pass records as Go structs)
– Protobuf producer (uses pb/flow.proto)
D. Formatter Layer
– JSON, text, or binary (via format/json, format/text, format/binary)
– Custom field mapping (cmd/goflow2/mapping.yaml)
E. Transport Layer
– File or stdout (transport/file)
– Kafka (transport/kafka using Sarama client)
F. Enricher (optional)
– cmd/enricher reads Protobuf, enriches via MaxMind DBs, outputs Protobuf
G. Metrics & Monitoring
– metrics exposed on HTTP (Prometheus endpoint, metrics/…)
H. External Services
– Kafka cluster
– File system or log aggregator (e.g. Loki, Splunk)
– Downstream storage/graphing: ClickHouse + Grafana or Elasticsearch + Kibana

##### Relationships and data flow
UDP Listeners → Decoder → Producer → Formatter → Transport → External Sink
- At each stage—Decoder, Producer, Transport—metrics are emitted to Prometheus.
- Enricher is a side-pipeline that consumes Protobuf output and feeds back into transport/formatter.
- Mapping component intercepts in Producer stage to apply custom field mappings.

##### Architectural patterns & design principles
- Pipeline/chain of responsibility: each stage implements a shared interface.
- Strategy pattern: formatters and transports are pluggable at runtime via CLI flags.
- Separation of concerns: decoding, producing, formatting, transport and enrichment are isolated packages.
- Modular, star-topology: central CLI orchestrates independent modules.

##### Essential details
- Ports (6343/2055 for UDP).
- Configurable mapping file for IPFIX templates.
- CLI flags that select strategies (-format, ‑transport).
- Protobuf schema version (pb/flow.proto).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Input Layer
    subgraph "1: Input Layer" 
        UDP["UDP Listeners\n(Ports 2055/6343)"]:::core
    end

    %% Decoder Layer
    subgraph "2: Decoder Layer" 
        NFIX["NetFlow/IPFIX Decoder v9"]:::core
        NLEG["NetFlow Legacy Decoder v5"]:::core
        SFLOW["sFlow Decoder v5"]:::core
        DUTIL["Decoder Utilities"]:::utils
    end

    %% Producer Layer
    subgraph "3: Producer Layer"
        RAW["Raw Go Struct Producer"]:::core
        PROTOP["Protobuf Producer\n(pb/flow.proto)"]:::core
        MAPP["Custom Mapping\n(mapping.yaml)"]:::optional
    end

    %% Formatter Layer
    subgraph "4: Formatter Layer"
        FJSON["JSON Formatter"]:::core
        FTEXT["Text Formatter"]:::core
        FBIN["Binary Formatter"]:::core
    end

    %% Transport Layer
    subgraph "5: Transport Layer"
        TFILE["File/Stdout Transport"]:::core
        TKAFKA["Kafka Transport"]:::core
        TINTF["Transport Interface"]:::utils
    end

    %% External Sinks
    subgraph "6: External Sinks"
        KCLUSTER["Kafka Cluster"]:::external
        FILES["File/Log Storage"]:::external
        CHGRA["ClickHouse + Grafana"]:::external
        ELK["Elasticsearch + Kibana"]:::external
    end

    %% Optional Enricher
    subgraph "Enricher (Optional)"
        ENRICH["Enricher CLI"]:::optional
        EPROTO["flowext.proto"]:::optional
    end

    %% Metrics & Monitoring
    PROM["Prometheus\nMetrics Endpoint"]:::external

    %% Data Flow
    UDP --> NFIX
    UDP --> NLEG
    UDP --> SFLOW

    NFIX --> RAW
    NLEG --> RAW
    SFLOW --> RAW

    RAW --> PROTOP
    RAW --> PROTOP
    PROTOP --> MAPP

    MAPP --> FJSON
    MAPP --> FTEXT
    MAPP --> FBIN

    FJSON --> TFILE
    FTEXT --> TFILE
    FBIN --> TFILE

    FJSON --> TKAFKA
    FTEXT --> TKAFKA
    FBIN --> TKAFKA

    TFILE --> FILES
    TKAFKA --> KCLUSTER

    KCLUSTER --> CHGRA
    FILES --> ELK

    %% Enricher flow
    PROTOP --> ENRICH
    ENRICH --> PROTOP

    %% Metrics Taps
    NFIX -.->|metrics| PROM
    NLEG -.->|metrics| PROM
    SFLOW -.->|metrics| PROM
    RAW -.->|metrics| PROM
    PROTOP -.->|metrics| PROM
    FJSON -.->|metrics| PROM
    FTEXT -.->|metrics| PROM
    FBIN -.->|metrics| PROM
    TFILE -.->|metrics| PROM
    TKAFKA -.->|metrics| PROM

    %% Example Pipelines
    subgraph "Example Compose Pipelines"
        ELKPIPE["ELK Pipeline\n(docker-compose.yml)"]:::external
        KCGPIPE["Kafka+ClickHouse+Grafana\n(docker-compose.yml)"]:::external
    end
    ELKPIPE --> ELK
    KCGPIPE --> CHGRA

    %% Click Events
    click UDP "https://github.com/netsampler/goflow2/blob/main/cmd/goflow2/main.go"
    click MAPP "https://github.com/netsampler/goflow2/blob/main/cmd/goflow2/mapping.yaml"
    click NFIX "https://github.com/netsampler/goflow2/tree/main/decoders/netflow/"
    click NLEG "https://github.com/netsampler/goflow2/tree/main/decoders/netflowlegacy/"
    click SFLOW "https://github.com/netsampler/goflow2/tree/main/decoders/sflow/"
    click DUTIL "https://github.com/netsampler/goflow2/tree/main/decoders/utils/"
    click RAW "https://github.com/netsampler/goflow2/blob/main/producer/raw/raw.go"
    click PROTOP "https://github.com/netsampler/goflow2/tree/main/producer/proto/"
    click FBIN "https://github.com/netsampler/goflow2/blob/main/format/binary/binary.go"
    click FJSON "https://github.com/netsampler/goflow2/blob/main/format/json/json.go"
    click FTEXT "https://github.com/netsampler/goflow2/blob/main/format/text/text.go"
    click TFILE "https://github.com/netsampler/goflow2/blob/main/transport/file/transport.go"
    click TKAFKA "https://github.com/netsampler/goflow2/blob/main/transport/kafka/kafka.go"
    click TINTF "https://github.com/netsampler/goflow2/blob/main/transport/transport.go"
    click ENRICH "https://github.com/netsampler/goflow2/blob/main/cmd/enricher/main.go"
    click EPROTO "https://github.com/netsampler/goflow2/blob/main/cmd/enricher/pb/flowext.proto"
    click PROM "https://github.com/netsampler/goflow2/blob/main/metrics/metrics.go"
    click ELKPIPE "https://github.com/netsampler/goflow2/blob/main/compose/elk/docker-compose.yml"
    click KCGPIPE "https://github.com/netsampler/goflow2/blob/main/compose/kcg/docker-compose.yml"

    %% Styles
    classDef core fill:#bbdefb,stroke:#1e88e5;
    classDef optional fill:#c8e6c9,stroke:#388e3c;
    classDef external fill:#ffe0b2,stroke:#f57c00;
    classDef utils fill:#e1bee7,stroke:#8e24aa;
```
