# phaag/nfdump architecture

> Architecture diagram of the GitHub repository phaag/nfdump, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phaag/nfdump
- Repository: https://github.com/phaag/nfdump

## Overview

##### Project type and purpose
- nfdump is a C-based, command-line suite for collecting, processing, filtering, aggregating and enriching NetFlow/IPFIX/sFlow data.
- It comprises multiple standalone tools (collectors, processors, utilities) that share core libraries for file I/O, flow parsing, filtering and enrichment.

##### File structure
- Top-level executables live under src/collector, src/nfcapd, src/sflow, src/nfpcapd, src/nfreader, src/nfdump, src/nfexpire, src/nfanon, src/nfreplay, etc.
- Core libraries in src/libnffile (disk format, compression, storage, file rotation) and src/libnfdump (flow data model, filter engine, enrichment modules).
- Inline performance modules in src/inline (hashing, I/O helpers, formatting).
- Output backends in src/output (CSV, JSON, raw, NDJSON, payload-specific).
- Optional enrichment plugins under src/libnfdump/{maxmind,tor,ja3,ja4,ssl,dns,filter,…}.
- Build and configuration scripts (autotools, Makefile.am, configure.ac).

##### Key insights from README
- Tools are invoked independently via command line.
- Collection (nfcapd, sfcapd, nfpcapd) → persistent rotate-on-time files.
- Processing (nfdump, nfanon, nfexpire, nfreplay, ft2nfdump, nfprofile) reads those files to filter, aggregate, enrich, convert or replay flows.
- Enrichment requires external databases (Maxmind GeoDB, TorDB).
- Outputs can feed monitoring stacks (InfluxDB, Prometheus via nfinflux/nfexporter).

##### Main components & interactions
A. Network sources (routers/switches exporting NetFlow/IPFIX/sFlow)
B. Collector daemons
- nfcapd: NetFlow collector
- sfcapd: sFlow collector
- nfpcapd: pcap-to-NetFlow converter
C. Storage layer
- libnffile: rotates & writes flow files to disk (optional compression via lz4, lzo, zstd, bzip2)
D. Processing & utility tools
- nfdump: filter & aggregate
- nfanon: anonymizer
- nfexpire: expiration manager
- nfreplay: replay flows
- ft2nfdump: format converter
- nfprofile/nftrack: NfSen plugins
E. Core libraries
- libnfdump: flow model, filter grammar, aggregation, digest (md5/sha256), parsing of NetFlow/sFlow/IPFIX protocols
- Inline modules: high-performance I/O, hashing, formatting
- Output backends: text, CSV, JSON, NDJSON, raw, payload-specific
F. Enrichment modules (under libnfdump)
- Maxmind geolocation
- Tor exit-node lookup
- JA3/JA4 fingerprinting
G. External systems
- Databases: Maxmind GeoDB, TorDB
- Time-series endpoints: InfluxDB, Prometheus (via nfinflux/nfexporter)
- Downstream monitoring stacks (Grafana, NfSen)

Flow of data:
1. Packet/flow export → Collector daemon
2. Collector writes to disk (libnffile)
3. Processing tool loads flow files via libnffile
4. Core library (libnfdump) parses, filters, aggregates, optionally calls enrichment modules
5. Output backend formats results and writes to stdout, file, or pushes to external system

##### Architectural patterns & design principles
- Modular CLI tools: each function is its own binary.
- Shared libraries provide reusability (libnffile for I/O, libnfdump for data model).
- Plugin-style enrichment: separate modules for GeoDB, TorDB, fingerprinting.
- Unix philosophy: small, composable tools connected by files or pipes.
- High-performance inline modules for critical paths (hashing, formatting).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources & Collectors"
        class DSCol data
        NetDevices["Network Devices"]:::data
        Nfcapd["nfcapd (NetFlow Collector)"]:::data
        Sfcapd["sfcapd (sFlow Collector)"]:::data
        Nfpcapd["nfpcapd (pcap→NetFlow)"]:::data
    end

    %% Storage Layer
    subgraph "Storage Layer"
        Libnffile["libnffile"]:::storage
    end

    %% Processing Tools
    subgraph "Processing & Utilities"
        Nfdump["nfdump"]:::processing
        Nfanon["nfanon"]:::processing
        Nfexpire["nfexpire"]:::processing
        Nfreplay["nfreplay"]:::processing
        Ft2nfdump["ft2nfdump"]:::processing
        Nfprofile["nfprofile"]:::processing
        Nftrack["nftrack"]:::processing
    end

    %% Shared Libraries
    subgraph "Shared Libraries"
        Libnfdump["libnfdump"]:::libs
        Inline["Inline Modules"]:::libs
        Output["Output Backends"]:::libs
    end

    %% Enrichment Modules
    subgraph "Enrichment Modules"
        Maxmind["Maxmind GeoDB"]:::enrich
        TorMod["Tor Exit-Node Lookup"]:::enrich
        JA3["JA3 Fingerprinting"]:::enrich
        JA4["JA4 Fingerprinting"]:::enrich
        SSLMod["SSL Enrichment"]:::enrich
        DNSMod["DNS Enrichment"]:::enrich
    end

    %% External Systems
    subgraph "External Systems & Outputs"
        GeoDB["Maxmind GeoDB"]:::external
        TorDB["TorDB"]:::external
        Influx["InfluxDB"]:::external
        Prometheus["Prometheus"]:::external
        Grafana["Grafana"]:::external
        NfSen["NfSen"]:::external
    end

    %% Data Flow
    NetDevices -->|"exports flows"| Nfcapd
    NetDevices -->|"exports flows"| Sfcapd
    NetDevices -->|"exports packets"| Nfpcapd

    Nfcapd -->|"writes files"| Libnffile
    Sfcapd -->|"writes files"| Libnffile
    Nfpcapd -->|"writes files"| Libnffile

    Libnffile -->|"reads files"| Nfdump
    Libnffile -->|"reads files"| Nfanon
    Libnffile -->|"reads files"| Nfexpire
    Libnffile -->|"reads files"| Nfreplay
    Libnffile -->|"reads files"| Ft2nfdump
    Libnffile -->|"reads files"| Nfprofile
    Libnffile -->|"reads files"| Nftrack

    Nfdump -->|"uses"| Libnfdump
    Nfanon -->|"uses"| Libnfdump
    Nfexpire -->|"uses"| Libnfdump
    Nfreplay -->|"uses"| Libnfdump
    Ft2nfdump -->|"uses"| Libnfdump
    Nfprofile -->|"uses"| Libnfdump
    Nftrack -->|"uses"| Libnfdump

    Libnfdump -->|"invokes"| Maxmind
    Libnfdump -->|"invokes"| TorMod
    Libnfdump -->|"invokes"| JA3
    Libnfdump -->|"invokes"| JA4
    Libnfdump -->|"invokes"| SSLMod
    Libnfdump -->|"invokes"| DNSMod

    Maxmind -->|"lookup"| GeoDB
    TorMod -->|"lookup"| TorDB

    Nfdump -->|"formats via"| Output
    Nfanon -->|"formats via"| Output
    Nfexpire -->|"formats via"| Output
    Nfreplay -->|"formats via"| Output
    Ft2nfdump -->|"formats via"| Output
    Nfprofile -->|"formats via"| Output
    Nftrack -->|"formats via"| Output

    Output -->|"push metrics"| Influx
    Output -->|"push metrics"| Prometheus
    Nfprofile -->|"plugin for"| NfSen
    Nftrack -->|"plugin for"| NfSen
    Influx -->|"visualize"| Grafana
    Prometheus -->|"visualize"| Grafana

    %% Click Events
    click Nfcapd "https://github.com/phaag/nfdump/blob/master/src/nfcapd/nfcapd.c"
    click Sfcapd "https://github.com/phaag/nfdump/blob/master/src/sflow/sfcapd.c"
    click Nfpcapd "https://github.com/phaag/nfdump/blob/master/src/nfpcapd/nfpcapd.c"
    click Libnffile "https://github.com/phaag/nfdump/tree/master/src/libnffile/"
    click Nfdump "https://github.com/phaag/nfdump/blob/master/src/nfdump/nfdump.c"
    click Nfanon "https://github.com/phaag/nfdump/blob/master/src/nfanon/nfanon.c"
    click Nfexpire "https://github.com/phaag/nfdump/blob/master/src/nfexpire/nfexpire.c"
    click Nfreplay "https://github.com/phaag/nfdump/blob/master/src/nfreplay/nfreplay.c"
    click Ft2nfdump "https://github.com/phaag/nfdump/blob/master/src/ft2nfdump/ft2nfdump.c"
    click Nfprofile "https://github.com/phaag/nfdump/blob/master/src/nfsen/nfprofile.c"
    click Nftrack "https://github.com/phaag/nfdump/blob/master/src/nfsen/nftrack.c"
    click Libnfdump "https://github.com/phaag/nfdump/tree/master/src/libnfdump/"
    click Inline "https://github.com/phaag/nfdump/tree/master/src/inline/"
    click Output "https://github.com/phaag/nfdump/tree/master/src/output/"
    click Maxmind "https://github.com/phaag/nfdump/tree/master/src/libnfdump/maxmind/"
    click TorMod "https://github.com/phaag/nfdump/tree/master/src/libnfdump/tor/"
    click JA3 "https://github.com/phaag/nfdump/tree/master/src/libnfdump/ja3/"
    click JA4 "https://github.com/phaag/nfdump/tree/master/src/libnfdump/ja4/"
    click SSLMod "https://github.com/phaag/nfdump/tree/master/src/libnfdump/ssl/"
    click DNSMod "https://github.com/phaag/nfdump/tree/master/src/libnfdump/dns/"

    %% Styles
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef storage fill:#eeeeee,stroke:#757575,color:#212121
    classDef processing fill:#c8e6c9,stroke:#388e3c,color:#1b5e20
    classDef libs fill:#f5f5f5,stroke:#9e9e9e,color:#424242
    classDef enrich fill:#ffe0b2,stroke:#fb8c00,color:#e65100
    classDef external fill:#e1bee7,stroke:#8e24aa,color:#4a148c
```
