# sau007/threat-intelligence-aggregator architecture

> Architecture diagram of the GitHub repository sau007/threat-intelligence-aggregator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sau007/threat-intelligence-aggregator
- Repository: https://github.com/sau007/threat-intelligence-aggregator

## Overview

##### What this project is (type + purpose)
From the README and file tree, this is a **single-process Python threat-intel enrichment/aggregation tool** (likely CLI-driven) that:
- Takes **indicators** (IP, domain, URL, file hash)
- Queries **external OSINT sources** (explicitly VirusTotal; potentially others)
- **Aggregates** results
- Produces a **risk score + verdict** for each indicator

This is not a multi-service web app; it’s best represented as a **standalone analyzer application** with clear internal modules and external API dependencies.

##### What the file tree implies about architecture
Top-level files indicate a small, direct architecture:

- `core.py`
The main logic entrypoint (likely: read inputs → call sources → score → output). This is the “application layer” plus potentially multiple responsibilities in one file.

- `config.json`

- `indicators.txt`
Input dataset. Indicates batch processing is likely.

- `requirements.txt`
External libraries (HTTP clients, VT SDK, etc.). Important for noting “integration mechanism” (REST over HTTPS).

- `README.md`, `README.txt`
Documentation; may hint intended usage (CLI invocation, environment needs).

##### Actors / Inputs / Outputs (outside the app boundary)
- **Threat Analyst / User**
- **Indicators Source**: `indicators.txt` (or stdin / file path)

##### Runner / Orchestrator (core entrypoint)
Coordinates the overall workflow.

##### Config Loader
Reads `config.json` (API keys, enabled sources, thresholds, timeouts, rate limits).

##### Indicator Ingestion + Normalization
- Reads `indicators.txt`
- Classifies indicator type (IP/domain/URL/hash)
- Normalizes format (e.g., strip whitespace, canonicalize URLs)

##### Enrichment Clients (OSINT adapters)
- **VirusTotal Client** (explicit)
Each client should be shown as an adapter that does **HTTP(S) REST calls** and returns structured results.

##### Aggregation / Correlation Engine
- Combines responses across sources
- Deduplicates and merges fields
- Handles partial failures/timeouts

##### Scoring + Verdict Engine
- Computes “risk score”
- Assigns verdict (e.g., benign/suspicious/malicious/unknown)

- **Logging**
- **Rate limiting / retry policy**

##### External Systems (explicit dependencies)
- **VirusTotal API** (external SaaS)
- Network boundary: **Internet**

##### For each indicator
- Normalize/classify
- Call VirusTotal (and other sources)
- Aggregate results
- Score + verdict
- Output/report

- User → Runner/Orchestrator (command invocation)
- Runner → Config Loader → `config.json`
- Runner → Indicator Ingestion → `indicators.txt`
- Indicator Ingestion → Enrichment Clients (per indicator)
- Enrichment Clients ↔ External OSINT APIs (HTTPS REST)
- Enrichment Clients → Aggregation Engine
- Aggregation Engine → Scoring/Verdict Engine
- Scoring/Verdict Engine → Reporting/Formatting → Output (console/file)

- “indicator batch”
- “normalized indicator”
- “API request/response”
- “enriched intel”
- “aggregated findings”
- “score/verdict”
- “final report”

##### Architectural patterns/design principles to highlight
Even if not formally implemented as separate classes, the architecture naturally matches:

- **Pipeline / staged processing**: ingest → enrich → aggregate → score → report
- **Adapter pattern for OSINT providers**: each source client isolates API-specific behavior
- **Configuration-driven behavior**: `config.json` influences enabled sources, thresholds, etc.
- **Fail-soft integration**: external APIs can fail; aggregation/scoring should handle missing signals

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Actors / Inputs / Outputs
  %% =========================
  subgraph "Actors / I/O (Outside App Boundary)"
    direction TB
    User["Threat Analyst / User"]:::actor
    IndicatorsFile[/"Indicators Input\n(indicators.txt)"/]:::artifact
    Output["Console Output / Report Output"]:::artifact
  end

  %% =========================
  %% Application Boundary
  %% =========================
  subgraph "Threat Intelligence Aggregator (Single Process Python App)"
    direction TB

    Orchestrator["Runner / Orchestrator\n(core.py entrypoint)"]:::app

    subgraph "Configuration Boundary"
      direction TB
      ConfigLoader["Config Loader"]:::app
      ConfigFile[/"Runtime Config\n(config.json)"/]:::artifact
    end

    subgraph "Pipeline (Staged Processing)"
      direction TB
      Ingestion["Indicator Ingestion + Normalization\n(read/classify/normalize)"]:::app

      subgraph "Enrichment Layer (OSINT Adapters)"
        direction TB
        VTClient["VirusTotal Client\n(HTTPS REST adapter)"]:::app
        OtherClients["Other OSINT Clients\n(extension point)"]:::appOptional
        RateRetry["Rate limiting / retries / timeouts\n(cross-cutting)"]:::crosscut
      end

      Aggregation["Aggregation / Correlation Engine\n(merge/dedupe/partial-failure handling)"]:::app
      Scoring["Risk Scoring + Verdict Engine"]:::app
      Reporting["Reporting / Formatting\n(build per-indicator output)"]:::app
    end

    Logging["Logging\n(cross-cutting)"]:::crosscut
    Caching["Caching\n(optional)"]:::crosscutOptional

    Deps["Dependencies / Integration Mechanism\n(requirements.txt)"]:::artifact
    Docs["Documentation / Usage\n(README)"]:::artifact
  end

  %% =========================
  %% External Systems
  %% =========================
  subgraph "External Systems (Internet)"
    direction TB
    VTAPI(("VirusTotal API\n(External SaaS)")):::external
    OtherOSINT(("Other OSINT APIs\n(Optional)")):::externalOptional
  end

  %% =========================
  %% Data / Control Flow
  %% =========================
  User -->|"invokes CLI"| Orchestrator

  Orchestrator -->|"load config"| ConfigLoader
  ConfigLoader -->|"read"| ConfigFile

  Orchestrator -->|"read indicator batch"| Ingestion
  IndicatorsFile -->|"indicator batch"| Ingestion

  Ingestion -->|"normalized indicator"| VTClient
  Ingestion -.->|"normalized indicator"| OtherClients

  VTClient -->|"API request"| VTAPI
  VTAPI -->|"API response"| VTClient

  OtherClients -.->|"API request"| OtherOSINT
  OtherOSINT -.->|"API response"| OtherClients

  VTClient -->|"enriched intel"| Aggregation
  OtherClients -.->|"enriched intel"| Aggregation

  Aggregation -->|"aggregated findings"| Scoring
  Scoring -->|"score + verdict"| Reporting
  Reporting -->|"final report"| Output

  %% Cross-cutting links
  Orchestrator -.->|"logs"| Logging
  Ingestion -.->|"logs"| Logging
  VTClient -.->|"logs"| Logging
  Aggregation -.->|"logs"| Logging
  Scoring -.->|"logs"| Logging
  Reporting -.->|"logs"| Logging

  RateRetry -.->|"applies to"| VTClient
  RateRetry -.->|"applies to"| OtherClients
  Caching -.->|"optional cache"| VTClient
  Caching -.->|"optional cache"| OtherClients

  Deps -.->|"provides HTTP/SDK libs"| VTClient
  Deps -.->|"provides HTTP/SDK libs"| OtherClients
  Docs -.->|"usage guides"| User

  %% =========================
  %% Click Events (component_mapping)
  %% =========================
  click Orchestrator "https://github.com/sau007/threat-intelligence-aggregator/blob/main/core.py"
  click ConfigLoader "https://github.com/sau007/threat-intelligence-aggregator/blob/main/core.py"
  click Ingestion "https://github.com/sau007/threat-intelligence-aggregator/blob/main/core.py"
  click VTClient "https://github.com/sau007/threat-intelligence-aggregator/blob/main/core.py"
  click OtherClients "https://github.com/sau007/threat-intelligence-aggregator/blob/main/core.py"
  click Aggregation "https://github.com/sau007/threat-intelligence-aggregator/blob/main/core.py"
  click Scoring "https://github.com/sau007/threat-intelligence-aggregator/blob/main/core.py"
  click Reporting "https://github.com/sau007/threat-intelligence-aggregator/blob/main/core.py"
  click ConfigFile "https://github.com/sau007/threat-intelligence-aggregator/blob/main/config.json"
  click IndicatorsFile "https://github.com/sau007/threat-intelligence-aggregator/blob/main/indicators.txt"
  click Deps "https://github.com/sau007/threat-intelligence-aggregator/blob/main/requirements.txt"
  click Docs "https://github.com/sau007/threat-intelligence-aggregator/blob/main/README.md"

  %% =========================
  %% Styles
  %% =========================
  classDef actor fill:#E8F1FF,stroke:#2B6CB0,stroke-width:2px,color:#0B1F3A
  classDef app fill:#D9ECFF,stroke:#1E4E8C,stroke-width:2px,color:#0B1F3A
  classDef appOptional fill:#EAF4FF,stroke:#2B6CB0,stroke-width:1px,stroke-dasharray:6 4,color:#0B1F3A
  classDef artifact fill:#F2F2F2,stroke:#6B7280,stroke-width:1.5px,color:#111827
  classDef external fill:#FFE6CC,stroke:#C05621,stroke-width:2px,color:#3B1D0A
  classDef externalOptional fill:#FFF2E5,stroke:#C05621,stroke-width:1.5px,stroke-dasharray:6 4,color:#3B1D0A
  classDef crosscut fill:#E9D8FD,stroke:#6B46C1,stroke-width:1.5px,color:#1A1033
  classDef crosscutOptional fill:#F3E8FF,stroke:#6B46C1,stroke-width:1.5px,stroke-dasharray:6 4,color:#1A1033
```
