How cisco-eti/sf-processor works
Project type and purpose
• This is an open-source edge analytics pipeline (sf-processor) written in Go, part of the SysFlow Telemetry framework.
• It ingests SysFlow event streams, processes/enriches them through configurable plugins, makes policy-based decisions, and exports results to various targets.
File structure
• Top-level “core” directory holds the engine logic, split into four major sub-components:
• cache (in-memory tables)
• flattener (converts raw events to flat records)
• policyengine (Applies policies: Falco, Sigma, custom)
• exporter (encodes and ships processed data)
• The “driver” directory implements two built-in data source drivers (file, streaming socket) and orchestrates pipeline construction.
• “plugins” houses example custom actions, handlers, processors illustrating the plugin interface.
• “resources” contains JSON/YAML configs for pipelines, policies, ECS mappings, etc.
• docs, scripts, Makefile, Dockerfile support build, test, CI, deployment.
README
• Confirms pipeline composed of drivers → plugins connected via Go channels → exporters.
• Highlights extensibility: users can write their own plugins or drivers.
• Lists supported export targets (file, Elasticsearch, syslog, terminal).
Main components & interactions (to show
a. Drivers (Data Sources)
• FileDriver
• StreamingDriver (Unix socket)
b. Pipeline Orchestrator (driver/pipeline)
• Reads JSON config → instantiates plugin chain → wires channels.
c. Plugin Chain (Go routines + channels)
1. Prefilter (core/policyengine/source/prefilter.go)
2. Flattener (core/flattener)
3. Policy Engine (core/policyengine)
4. Processor (core/processor)
5. Exporter (core/exporter)
d. Exporter Internals
• Encoders (JSON, Avro, ECS)
• Transports (file, elastic, syslog, terminal, null)
e. Configuration & Resources
• Config JSON (resources/pipelines/*.json)
• Policy YAML (resources/policies/)
• ECS mapping (resources/mappings/ecs_mapping.json)
f. External Targets
• Elasticsearch cluster
• Filesystem (S3, local files)
• Syslog endpoint
• Console/STDOUT
Architectural patterns & design principles
• Pipeline pattern: Each plugin stage is a Go routine communicating via typed channels.
• Plugin interface: Stateless, composable units that implement a common Process/Handle signature.
• Separation of concerns: Drivers, core processing, exporting are decoupled.
• Extensibility: New drivers or plugins drop-in via JSON config.
• Concurrency: Each stage can run with a configurable worker pool.
Elements
• Clear labels: driver/file, driver/streaming, pipeline manager, each plugin by name.
• Concurrency note: annotate that each plugin stage is a pool of Go routines.
• Key technologies: Go, Apache Avro, JSON, UNIX socket, Elasticsearch.