# cisco-eti/sf-processor architecture

> Architecture diagram of the GitHub repository cisco-eti/sf-processor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cisco-eti/sf-processor
- Repository: https://github.com/cisco-eti/sf-processor

## Overview

##### Project type and purpose
- This is an open-source edge analytics pipeline (sf-processor) written in Go, part of the SysFlow Telemetry framework.
- It ingests SysFlow event streams, processes/enriches them through configurable plugins, makes policy-based decisions, and exports results to various targets.

##### File structure
- Top-level “core” directory holds the engine logic, split into four major sub-components:
- cache (in-memory tables)
- flattener (converts raw events to flat records)
- policyengine (Applies policies: Falco, Sigma, custom)
- exporter (encodes and ships processed data)
- The “driver” directory implements two built-in data source drivers (file, streaming socket) and orchestrates pipeline construction.
- “plugins” houses example custom actions, handlers, processors illustrating the plugin interface.
- “resources” contains JSON/YAML configs for pipelines, policies, ECS mappings, etc.
- docs, scripts, Makefile, Dockerfile support build, test, CI, deployment.

##### README
- Confirms pipeline composed of drivers → plugins connected via Go channels → exporters.
- Highlights extensibility: users can write their own plugins or drivers.
- Lists supported export targets (file, Elasticsearch, syslog, terminal).

##### Main components & interactions (to show
a. **Drivers (Data Sources)**
- FileDriver
- StreamingDriver (Unix socket)
b. **Pipeline Orchestrator (driver/pipeline)**
- Reads JSON config → instantiates plugin chain → wires channels.
c. **Plugin Chain (Go routines + channels)**
1. Prefilter (core/policyengine/source/prefilter.go)
2. Flattener (core/flattener)
3. Policy Engine (core/policyengine)
4. Processor (core/processor)
5. Exporter (core/exporter)
d. **Exporter Internals**
- Encoders (JSON, Avro, ECS)
- Transports (file, elastic, syslog, terminal, null)
e. **Configuration & Resources**
- Config JSON (resources/pipelines/*.json)
- Policy YAML (resources/policies/)
- ECS mapping (resources/mappings/ecs_mapping.json)
f. **External Targets**
- Elasticsearch cluster
- Filesystem (S3, local files)
- Syslog endpoint
- Console/STDOUT

##### Architectural patterns & design principles
- **Pipeline pattern**: Each plugin stage is a Go routine communicating via typed channels.
- **Plugin interface**: Stateless, composable units that implement a common Process/Handle signature.
- **Separation of concerns**: Drivers, core processing, exporting are decoupled.
- **Extensibility**: New drivers or plugins drop-in via JSON config.
- **Concurrency**: Each stage can run with a configurable worker pool.

##### Elements
- **Clear labels**: driver/file, driver/streaming, pipeline manager, each plugin by name.
- **Concurrency note**: annotate that each plugin stage is a pool of Go routines.
- **Key technologies**: Go, Apache Avro, JSON, UNIX socket, Elasticsearch.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Drivers Layer
    subgraph "Drivers" 
        FD["FileDriver"]:::driver
        SD["StreamingDriver"]:::driver
    end

    %% Pipeline Orchestrator
    PO["Pipeline Orchestrator"]:::core

    %% Configuration & Resources
    Config["Config JSON & Resources"]:::config

    %% Plugins Directory (Extensibility)
    Plugins["Plugins Directory"]:::config

    %% Plugin Chain
    subgraph "Plugin Chain" 
        PF["Prefilter"]:::plugin
        FL["Flattener"]:::plugin
        PE["PolicyEngine"]:::plugin
        PR["Processor"]:::plugin
        EX["Exporter"]:::plugin
    end

    %% Exporter Internals
    subgraph "Exporter Internals"
        JSONE["JSON Encoder"]:::exporter
        AVRO["Avro Encoder"]:::exporter
        ECSE["ECS Encoder"]:::exporter
        subgraph "Transports"
            T_FILE["File"]:::exporter
            T_ES["Elasticsearch"]:::exporter
            T_SYS["Syslog"]:::exporter
            T_TERM["Terminal"]:::exporter
            T_NULL["Null"]:::exporter
        end
    end

    %% External Targets
    subgraph "External Targets"
        ESCluster["Elasticsearch Cluster"]:::external
        FileSystem["Filesystem (S3/Local)"]:::external
        SyslogEP["Syslog Endpoint"]:::external
        Console["Console/STDOUT"]:::external
    end

    %% Data Flow
    FD -->|"SysFlow frames"| PO
    SD -->|"SysFlow frames"| PO
    Config -.->|"loads config"| PO
    Plugins -.->|"custom plugins"| PO

    PO -->|"frames"| PF
    PF -->|"FlatRecord"| FL
    FL -->|"FlatRecord"| PE
    PE -->|"EnforcementAction"| PR
    PR -->|"ProcessedRecord"| EX

    EX -->|"payload"| JSONE
    EX -->|"payload"| AVRO
    EX -->|"payload"| ECSE

    JSONE -->|"encoded payload"| T_FILE
    JSONE -->|"encoded payload"| T_ES
    JSONE -->|"encoded payload"| T_SYS
    JSONE -->|"encoded payload"| T_TERM
    JSONE -->|"encoded payload"| T_NULL

    AVRO -->|"encoded payload"| T_FILE
    AVRO -->|"encoded payload"| T_ES
    AVRO -->|"encoded payload"| T_SYS
    AVRO -->|"encoded payload"| T_TERM
    AVRO -->|"encoded payload"| T_NULL

    ECSE -->|"encoded payload"| T_FILE
    ECSE -->|"encoded payload"| T_ES
    ECSE -->|"encoded payload"| T_SYS
    ECSE -->|"encoded payload"| T_TERM
    ECSE -->|"encoded payload"| T_NULL

    T_FILE -->|"writes"| FileSystem
    T_ES -->|"indexes"| ESCluster
    T_SYS -->|"sends"| SyslogEP
    T_TERM -->|"prints"| Console
    T_NULL -->|"drops"| Console

    %% Click Events
    click FD "https://github.com/cisco-eti/sf-processor/blob/master/driver/sysflow/filedriver.go"
    click SD "https://github.com/cisco-eti/sf-processor/blob/master/driver/sysflow/streamingdriver.go"
    click PO "https://github.com/cisco-eti/sf-processor/blob/master/driver/pipeline/pipeline.go"
    click PO "https://github.com/cisco-eti/sf-processor/blob/master/driver/pipeline/config.go"
    click PO "https://github.com/cisco-eti/sf-processor/blob/master/driver/pipeline/plugincache.go"
    click PF "https://github.com/cisco-eti/sf-processor/blob/master/core/policyengine/source/flatrecord/prefilter.go"
    click FL "https://github.com/cisco-eti/sf-processor/blob/master/core/flattener/flattener.go"
    click FL "https://github.com/cisco-eti/sf-processor/blob/master/core/flattener/config.go"
    click FL "https://github.com/cisco-eti/sf-processor/blob/master/core/flattener/filter.go"
    click PE "https://github.com/cisco-eti/sf-processor/blob/master/core/policyengine/policyengine.go"
    click PE "https://github.com/cisco-eti/sf-processor/blob/master/core/policyengine/policyengine_flatrecord.go"
    click EX "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/exporter.go"
    click EX "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/commons/config.go"
    click JSONE "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/encoders/json.go"
    click AVRO "https://github.com/cisco-eti/sf-processor/tree/master/core/exporter/encoders/avro/avdl"
    click ECSE "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/encoders/ecs.go"
    click T_FILE "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/transports/file.go"
    click T_ES "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/transports/elastic.go"
    click T_SYS "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/transports/syslog.go"
    click T_TERM "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/transports/terminal.go"
    click T_NULL "https://github.com/cisco-eti/sf-processor/blob/master/core/exporter/transports/null.go"
    click Config "https://github.com/cisco-eti/sf-processor/tree/master/resources/pipelines"
    click Config "https://github.com/cisco-eti/sf-processor/tree/master/resources/policies"
    click Config "https://github.com/cisco-eti/sf-processor/blob/master/resources/mappings/ecs_mapping.json"
    click Plugins "https://github.com/cisco-eti/sf-processor/tree/master/plugins/actions/example/"
    click Plugins "https://github.com/cisco-eti/sf-processor/tree/master/plugins/handlers/printer/"
    click Plugins "https://github.com/cisco-eti/sf-processor/tree/master/plugins/processors/example/"

    %% Styling
    classDef core fill:#BBDEFB,stroke:#0D47A1,color:#0D47A1;
    classDef driver fill:#FFF59D,stroke:#F9A825,color:#F57F17;
    classDef plugin fill:#C8E6C9,stroke:#2E7D32,color:#2E7D32;
    classDef exporter fill:#FFE0B2,stroke:#E65100,color:#E65100;
    classDef config fill:#ECEFF1,stroke:#607D8B,color:#455A64,stroke-dasharray: 5 5;
    classDef external fill:#FFCCBC,stroke:#BF360C,color:#BF360C;
```
