How phaag/nfdump works
Project type and purpose
• nfdump is a C-based, command-line suite for collecting, processing, filtering, aggregating and enriching NetFlow/IPFIX/sFlow data.
• It comprises multiple standalone tools (collectors, processors, utilities) that share core libraries for file I/O, flow parsing, filtering and enrichment.
File structure
• Top-level executables live under src/collector, src/nfcapd, src/sflow, src/nfpcapd, src/nfreader, src/nfdump, src/nfexpire, src/nfanon, src/nfreplay, etc.
• Core libraries in src/libnffile (disk format, compression, storage, file rotation) and src/libnfdump (flow data model, filter engine, enrichment modules).
• Inline performance modules in src/inline (hashing, I/O helpers, formatting).
• Output backends in src/output (CSV, JSON, raw, NDJSON, payload-specific).
• Optional enrichment plugins under src/libnfdump/{maxmind,tor,ja3,ja4,ssl,dns,filter,…}.
• Build and configuration scripts (autotools, Makefile.am, configure.ac).
Key insights from README
• Tools are invoked independently via command line.
• Collection (nfcapd, sfcapd, nfpcapd) → persistent rotate-on-time files.
• Processing (nfdump, nfanon, nfexpire, nfreplay, ft2nfdump, nfprofile) reads those files to filter, aggregate, enrich, convert or replay flows.
• Enrichment requires external databases (Maxmind GeoDB, TorDB).
• Outputs can feed monitoring stacks (InfluxDB, Prometheus via nfinflux/nfexporter).
Main components & interactions
A. Network sources (routers/switches exporting NetFlow/IPFIX/sFlow)
B. Collector daemons
• nfcapd: NetFlow collector
• sfcapd: sFlow collector
• nfpcapd: pcap-to-NetFlow converter
C. Storage layer
• libnffile: rotates & writes flow files to disk (optional compression via lz4, lzo, zstd, bzip2)
D. Processing & utility tools
• nfdump: filter & aggregate
• nfanon: anonymizer
• nfexpire: expiration manager
• nfreplay: replay flows
• ft2nfdump: format converter
• nfprofile/nftrack: NfSen plugins
E. Core libraries
• libnfdump: flow model, filter grammar, aggregation, digest (md5/sha256), parsing of NetFlow/sFlow/IPFIX protocols
• Inline modules: high-performance I/O, hashing, formatting
• Output backends: text, CSV, JSON, NDJSON, raw, payload-specific
F. Enrichment modules (under libnfdump)
• Maxmind geolocation
• Tor exit-node lookup
• JA3/JA4 fingerprinting
G. External systems
• Databases: Maxmind GeoDB, TorDB
• Time-series endpoints: InfluxDB, Prometheus (via nfinflux/nfexporter)
• Downstream monitoring stacks (Grafana, NfSen)
Flow of data:
1. Packet/flow export → Collector daemon
2. Collector writes to disk (libnffile)
3. Processing tool loads flow files via libnffile
4. Core library (libnfdump) parses, filters, aggregates, optionally calls enrichment modules
5. Output backend formats results and writes to stdout, file, or pushes to external system
Architectural patterns & design principles
• Modular CLI tools: each function is its own binary.
• Shared libraries provide reusability (libnffile for I/O, libnfdump for data model).
• Plugin-style enrichment: separate modules for GeoDB, TorDB, fingerprinting.
• Unix philosophy: small, composable tools connected by files or pipes.
• High-performance inline modules for critical paths (hashing, formatting).