# pedramamini/threatingestor architecture

> Architecture diagram of the GitHub repository pedramamini/threatingestor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pedramamini/threatingestor
- Repository: https://github.com/pedramamini/threatingestor

## Overview

ThreatIngestor polls configured threat-intelligence sources, extracts and normalizes indicators, applies filtering/state, then dispatches artifacts to configured operators. The README documents a broad plugin catalog; source and operator families are grouped where their individual implementations were not sampled. Edges reflect the main orchestration and the explicitly shown extraction, persistence, and integration behavior.

Main entry points: [`threatingestor/__init__.py`](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/__init__.py) (Ingestor), [`threatingestor/sources`](https://github.com/pedramamini/threatingestor/tree/master/threatingestor/sources) (Feed Sources), [`threatingestor/sources/__init__.py`](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/__init__.py) (IOC Extraction), [`threatingestor/operators`](https://github.com/pedramamini/threatingestor/tree/master/threatingestor/operators) (Operator Plugins)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_cli["CLI Entry<br/>[__init__.py]"]
  node_config["Configuration<br/>[config.py]"]
  node_ingestor["Ingestor<br/>[__init__.py]"]
  node_state[("Source State<br/>[state.py]")]
  node_whitelist["Whitelist Filter<br/>[whitelist.py]"]
end

subgraph group_ingestion["Source Plugins"]
  node_sources["Feed Sources"]
  node_rss["RSS &amp; Sitemap<br/>[rss.py]"]
  node_web["Web &amp; Image<br/>[web.py]"]
  node_code["Git &amp; GitHub<br/>[github.py]"]
  node_social["Twitter &amp; VirusTotal<br/>[twitter.py]"]
  node_queue_source["SQS Source<br/>[sqs.py]"]
end

subgraph group_processing["Artifact Processing"]
  node_extract["IOC Extraction<br/>[__init__.py]"]
  node_artifacts["Artifact Types<br/>[artifacts.py]"]
end

subgraph group_outputs["Operators &amp; Extras"]
  node_operators["Operator Plugins"]
  node_file_db["CSV &amp; SQL<br/>[csv.py]"]
  node_threat_platforms["MISP &amp; ThreatKB<br/>[misp.py]"]
  node_queue_operator["Queue Operators<br/>[sqs.py]"]
  node_queue_worker["Queue Worker<br/>[queueworker.py]"]
  node_webapp["Artifact Browser<br/>[webapp.py]"]
end

node_user(("Operator"))
node_source_services(("Threat Feeds"))
node_destinations(("External Systems"))
node_artifact_db[("Artifact SQLite DB")]
node_queues["SQS / Beanstalk"]

node_user -->|"starts"| node_cli
node_cli -->|"creates"| node_ingestor
node_ingestor -->|"loads"| node_config
node_ingestor -->|"runs plugins"| node_sources
node_ingestor -->|"reads and saves"| node_state
node_ingestor -->|"filters artifacts"| node_whitelist
node_sources -->|"fetches content"| node_source_services
node_rss -->|"processes entries"| node_extract
node_extract -->|"creates"| node_artifacts
node_ingestor -->|"dispatches artifacts"| node_operators
node_operators -->|"handles"| node_artifacts
node_file_db -->|"writes records"| node_destinations
node_threat_platforms -->|"updates events"| node_destinations
node_queue_operator -->|"sends messages"| node_queues
node_queue_source -->|"reads messages"| node_queues
node_queue_worker -.->|"reads and writes"| node_queues
node_webapp -.->|"queries"| node_artifact_db

click node_cli "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/__init__.py"
click node_config "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/config.py"
click node_ingestor "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/__init__.py"
click node_state "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/state.py"
click node_whitelist "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/whitelist.py"
click node_sources "https://github.com/pedramamini/threatingestor/tree/master/threatingestor/sources"
click node_rss "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/rss.py"
click node_web "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/web.py"
click node_code "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/github.py"
click node_social "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/twitter.py"
click node_queue_source "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/sqs.py"
click node_extract "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/__init__.py"
click node_artifacts "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/artifacts.py"
click node_operators "https://github.com/pedramamini/threatingestor/tree/master/threatingestor/operators"
click node_file_db "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/operators/csv.py"
click node_threat_platforms "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/operators/misp.py"
click node_queue_operator "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/operators/sqs.py"
click node_queue_worker "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/extras/queueworker.py"
click node_webapp "https://github.com/pedramamini/threatingestor/blob/master/threatingestor/extras/webapp.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_ingestor,node_state,node_whitelist toneBlue
class node_sources,node_rss,node_web,node_code,node_social,node_queue_source,node_artifact_db toneAmber
class node_extract,node_artifacts toneMint
class node_operators,node_file_db,node_threat_platforms,node_queue_operator,node_queue_worker,node_webapp toneRose
class node_user,node_source_services,node_destinations,node_queues toneIndigo
```

## Components

### Orchestration

- [CLI Entry](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/__init__.py): `threatingestor/__init__.py`
- [Configuration](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/config.py): `threatingestor/config.py`
- [Ingestor](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/__init__.py): `threatingestor/__init__.py`
- [Source State](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/state.py): `threatingestor/state.py`
- [Whitelist Filter](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/whitelist.py): `threatingestor/whitelist.py`

### Source Plugins

- [Feed Sources](https://github.com/pedramamini/threatingestor/tree/master/threatingestor/sources): `threatingestor/sources`
- [RSS & Sitemap](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/rss.py): `threatingestor/sources/rss.py`
- [Web & Image](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/web.py): `threatingestor/sources/web.py`
- [Git & GitHub](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/github.py): `threatingestor/sources/github.py`
- [Twitter & VirusTotal](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/twitter.py): `threatingestor/sources/twitter.py`
- [SQS Source](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/sqs.py): `threatingestor/sources/sqs.py`

### Artifact Processing

- [IOC Extraction](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/sources/__init__.py): `threatingestor/sources/__init__.py`
- [Artifact Types](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/artifacts.py): `threatingestor/artifacts.py`

### Operators & Extras

- [Operator Plugins](https://github.com/pedramamini/threatingestor/tree/master/threatingestor/operators): `threatingestor/operators`
- [CSV & SQL](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/operators/csv.py): `threatingestor/operators/csv.py`
- [MISP & ThreatKB](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/operators/misp.py): `threatingestor/operators/misp.py`
- [Queue Operators](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/operators/sqs.py): `threatingestor/operators/sqs.py`
- [Queue Worker](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/extras/queueworker.py): `threatingestor/extras/queueworker.py`
- [Artifact Browser](https://github.com/pedramamini/threatingestor/blob/master/threatingestor/extras/webapp.py): `threatingestor/extras/webapp.py`

### Other components

- **Operator**
- **Threat Feeds**
- **External Systems**
- **Artifact SQLite DB**
- **SQS / Beanstalk**

## Connections

- Operator → CLI Entry: starts
- CLI Entry → Ingestor: creates
- Ingestor → Configuration: loads
- Ingestor → Feed Sources: runs plugins
- Ingestor → Source State: reads and saves
- Ingestor → Whitelist Filter: filters artifacts
- Feed Sources → Threat Feeds: fetches content
- RSS & Sitemap → IOC Extraction: processes entries
- IOC Extraction → Artifact Types: creates
- Ingestor → Operator Plugins: dispatches artifacts
- Operator Plugins → Artifact Types: handles
- CSV & SQL → External Systems: writes records
- MISP & ThreatKB → External Systems: updates events
- Queue Operators → SQS / Beanstalk: sends messages
- SQS Source → SQS / Beanstalk: reads messages
- Queue Worker → SQS / Beanstalk: reads and writes
- Artifact Browser → Artifact SQLite DB: queries