How sau007/threat-intelligence-aggregator works
What this project is (type + purpose)
From the README and file tree, this is a single-process Python threat-intel enrichment/aggregation tool (likely CLI-driven) that:
• Takes indicators (IP, domain, URL, file hash)
• Queries external OSINT sources (explicitly VirusTotal; potentially others)
• Aggregates results
• Produces a risk score + verdict for each indicator
This is not a multi-service web app; it’s best represented as a standalone analyzer application with clear internal modules and external API dependencies.
What the file tree implies about architecture
Top-level files indicate a small, direct architecture:
• core.py
The main logic entrypoint (likely: read inputs → call sources → score → output). This is the “application layer” plus potentially multiple responsibilities in one file.
• config.json
• indicators.txt
Input dataset. Indicates batch processing is likely.
• requirements.txt
External libraries (HTTP clients, VT SDK, etc.). Important for noting “integration mechanism” (REST over HTTPS).
• README.md, README.txt
Documentation; may hint intended usage (CLI invocation, environment needs).
Actors / Inputs / Outputs (outside the app boundary)
• Threat Analyst / User
• Indicators Source: indicators.txt (or stdin / file path)
Runner / Orchestrator (core entrypoint)
Coordinates the overall workflow.
Config Loader
Reads config.json (API keys, enabled sources, thresholds, timeouts, rate limits).
Indicator Ingestion + Normalization
• Reads indicators.txt
• Classifies indicator type (IP/domain/URL/hash)
• Normalizes format (e.g., strip whitespace, canonicalize URLs)
Enrichment Clients (OSINT adapters)
• VirusTotal Client (explicit)
Each client should be shown as an adapter that does HTTP(S) REST calls and returns structured results.
Aggregation / Correlation Engine
• Combines responses across sources
• Deduplicates and merges fields
• Handles partial failures/timeouts
Scoring + Verdict Engine
• Computes “risk score”
• Assigns verdict (e.g., benign/suspicious/malicious/unknown)
• Logging
• Rate limiting / retry policy
External Systems (explicit dependencies)
• VirusTotal API (external SaaS)
• Network boundary: Internet
For each indicator
• Normalize/classify
• Call VirusTotal (and other sources)
• Aggregate results
• Score + verdict
• Output/report
• User → Runner/Orchestrator (command invocation)
• Runner → Config Loader → config.json
• Runner → Indicator Ingestion → indicators.txt
• Indicator Ingestion → Enrichment Clients (per indicator)
• Enrichment Clients ↔ External OSINT APIs (HTTPS REST)
• Enrichment Clients → Aggregation Engine
• Aggregation Engine → Scoring/Verdict Engine
• Scoring/Verdict Engine → Reporting/Formatting → Output (console/file)
• “indicator batch”
• “normalized indicator”
• “API request/response”
• “enriched intel”
• “aggregated findings”
• “score/verdict”
• “final report”
Architectural patterns/design principles to highlight
Even if not formally implemented as separate classes, the architecture naturally matches:
• Pipeline / staged processing: ingest → enrich → aggregate → score → report
• Adapter pattern for OSINT providers: each source client isolates API-specific behavior
• Configuration-driven behavior: config.json influences enabled sources, thresholds, etc.
• Fail-soft integration: external APIs can fail; aggregation/scoring should handle missing signals