# oxapavan/ioc-extraction-through-n8n-automation architecture

> Architecture diagram of the GitHub repository oxapavan/ioc-extraction-through-n8n-automation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oxapavan/ioc-extraction-through-n8n-automation
- Repository: https://github.com/oxapavan/ioc-extraction-through-n8n-automation

## Overview

##### Project type and purpose
- This is an event-driven integration/automation pipeline built on n8n.
- Its goal is to periodically fetch cyber threat articles via RSS, extract IOCs (domains, hashes, IPs, CVEs) using a third-party API, reformat them, and send them into a SIEM for automated threat detection.

##### “file_tree” and README
- Only two top-level entries: README.md (detailed workflow description) and assets/IOC.png (example flowchart).
- No code directories—this is a pure n8n workflow definition.
- All logic lives as individual n8n nodes configured via the UI or JSON.

##### Main components
a. n8n Server / Workflow Engine (logical container)
b. Cron Trigger Node (scheduler)
c. RSS Fetch Nodes (threatpost, bleepingcomputer, Krebs, TheHackerNews, SecurityWeek)
d. Split-In-Batches Nodes (one per feed)
e. Code Nodes (JavaScript snippets to extract URLs)
f. Combine & Aggregate URLs Nodes
g. Loop / Iterator Node (to process each URL)
h. Format URL Payload Node
i. IOC Parser API Node (HTTP Request to external parsing service)
j. Format IOC Output Node (structure results for SIEM)
k. SIEM Integration Node (HTTP Request or Syslog to Splunk/ELK/QRadar)
l. External Services:
- RSS feed providers (public threat-intel sites)
- IOC Parser API endpoint
- SIEM ingestion API endpoint

##### Relationships and data flow
- Cron Trigger → each RSS Fetch → corresponding Split Node → Code Node → Combine URLs
- Combined URLs → Aggregate → Loop → Format Payload → IOC Parser API → Format IOC Output → SIEM
- External RSS feeds as upstream data sources feeding into the n8n “RSS Fetch” components
- Two downstream external services (IOC Parser API, SIEM)

##### Architectural patterns and design principles
- ETL / data-pipeline pattern: extract (RSS), transform (split, code, format), load (SIEM)
- Event-driven scheduling via Cron
- Micro-service calls via HTTP (to IOC Parser API and SIEM)
- Modular, extensible node-based design: easy to add new RSS feeds or output targets

##### Technologies and frameworks
- n8n workflow engine (Node.js-based)
- Cron scheduling (built-in n8n trigger)
- JavaScript (for Code nodes)
- HTTP Request nodes (for external API calls)
- SIEM ingestion protocols (HTTP HEC for Splunk, REST for ELK, syslog for QRadar)

##### Detail level
- Every distinct node or logical group of nodes
- Retry or error-handling patterns if configured (e.g., n8n error‐trigger nodes)

## Diagram (Mermaid)

```mermaid
flowchart TB
    classDef externalInput fill:#FFA500,stroke:#333
    classDef internal fill:#ADD8E6,stroke:#333
    classDef externalOutput fill:#90EE90,stroke:#333
    classDef docs fill:#D3D3D3,stroke:#333

    subgraph External_Feeds
        direction TB
        Threatpost["Threatpost RSS"]:::externalInput
        Bleeping["BleepingComputer RSS"]:::externalInput
        Krebs["KrebsOnSecurity RSS"]:::externalInput
        THN["TheHackerNews RSS"]:::externalInput
        SecWeek["SecurityWeek RSS"]:::externalInput
    end

    subgraph "n8n Server"
        direction TB
        Cron["Cron Trigger"]:::internal
        RSS1["RSS Fetch: Threatpost"]:::internal
        RSS2["RSS Fetch: BleepingComputer"]:::internal
        RSS3["RSS Fetch: KrebsOnSecurity"]:::internal
        RSS4["RSS Fetch: TheHackerNews"]:::internal
        RSS5["RSS Fetch: SecurityWeek"]:::internal
        Split1["SplitInBatches"]:::internal
        Split2["SplitInBatches"]:::internal
        Split3["SplitInBatches"]:::internal
        Split4["SplitInBatches"]:::internal
        Split5["SplitInBatches"]:::internal
        CodeExtract["Code Extract URLs"]:::internal
        Combine["Combine URLs"]:::internal
        Aggregate["Aggregate URLs"]:::internal
        Loop["Loop Iterator"]:::internal
        FormatPayload["Format URL Payload"]:::internal
        HTTPParser["HTTP Request: IOC Parser API"]:::internal
        FormatIOC["Format IOC Output"]:::internal
        SIEMNode["HTTP Request: SIEM Ingestion"]:::internal
    end

    subgraph External_Services
        direction TB
        IOCAPI["IOC Parser API Endpoint"]:::externalOutput
        SIEM["SIEM API/Syslog Endpoint"]:::externalOutput
    end

    Threatpost -->|fetch| RSS1
    Bleeping -->|fetch| RSS2
    Krebs -->|fetch| RSS3
    THN -->|fetch| RSS4
    SecWeek -->|fetch| RSS5

    Cron --> RSS1
    Cron --> RSS2
    Cron --> RSS3
    Cron --> RSS4
    Cron --> RSS5

    RSS1 --> Split1
    RSS2 --> Split2
    RSS3 --> Split3
    RSS4 --> Split4
    RSS5 --> Split5

    Split1 --> CodeExtract
    Split2 --> CodeExtract
    Split3 --> CodeExtract
    Split4 --> CodeExtract
    Split5 --> CodeExtract

    CodeExtract --> Combine
    Combine --> Aggregate
    Aggregate --> Loop
    Loop --> FormatPayload
    FormatPayload --> HTTPParser
    HTTPParser --> FormatIOC
    FormatIOC --> SIEMNode

    HTTPParser -->|parse| IOCAPI
    SIEMNode -->|ingest| SIEM

    Docs["README (Workflow Definition)"]:::docs
    ExampleDiagram["Example Workflow Diagram"]:::docs

    Docs -.-> Cron
    ExampleDiagram -.-> Combine

    click Docs "https://github.com/oxapavan/ioc-extraction-through-n8n-automation/blob/main/README.md"
    click ExampleDiagram "https://github.com/oxapavan/ioc-extraction-through-n8n-automation/blob/main/assets/IOC.png"
```
