How mhmdsan7/ioc_gatherer works
Project type and purpose
• This is a full-stack, web-based threat-intelligence tool built with Flask.
• Purpose: gather Indicators of Compromise (IoCs) from multiple external feeds, display results in the browser, and export reports to Excel.
File structure
• app.py: the main Flask application, defines UI routes and REST endpoints (/api/search, /api/export).
• templates/index.html: single-page web UI.
• requirements.txt: lists Python dependencies (Flask, Flask-CORS, requests, pandas, openpyxl, python-dotenv).
• .env.template: placeholder for API keys (AbuseIPDB, VirusTotal, ThreatFox, NIST NVD) loaded via python-dotenv.
• Dockerfile + docker-compose.yaml: containerize the Flask app and expose port 5000; mount a volume for exported files; manage container lifecycle and health checks.
README for additional insights
• Multi-feed support, IoC analysis across four types (IP, domain, hash, CVE).
• External intelligence sources accessed via HTTP.
• Excel export: pandas + openpyxl produce formatted workbooks.
• Two deployment modes: local venv and Docker (with compose orchestration).
a. Main components (container-level view)
1. User Browser (React-style single page? Just HTML+JS in index.html)
2. Flask Web Server Container
• HTTP Server & Routing
• UI Controller (renders index.html)
• API Layer
– Search Service (aggregates IoCs across feeds)
– Export Service (formats data into Excel)
• Service Clients for each external API:
– AbuseIPDB Client
– VirusTotal Client
– ThreatFox Client
– NVD Client
• Data Processor (pandas pipeline)
• Configuration Loader (python-dotenv picks up .env keys)
3. Volume Storage (persistent volume for generated Excel files)
4. External Threat Intelligence APIs (4 services accessed over HTTPS)
5. Docker / Docker-Compose (hosts the Flask container, sets up env vars, health checks)
b. Relationships & data flows
• Browser → Flask UI Controller (HTTP GET) → index.html
• Browser → Flask API Layer (HTTP POST /api/search) →
• Passes feed definitions → Search Service → iterates over feeds → calls each Service Client → returns aggregated results → JSON response to browser
• Browser → Flask API Layer (HTTP POST /api/export) →
• Passes JSON results → Export Service → Data Processor (pandas → openpyxl) → writes to Volume Storage → returns download link or file stream
• Flask reads .env for API keys at startup
• Docker-Compose injects environment, maps port 5000, mounts storage volume
c. Architectural patterns & principles
• MVC-like separation: controllers (routes) vs. services (external API clients & data processing)
• Single-container micro-service, no internal database—stateless except for temp Excel files
• Configuration as code (.env, Docker-compose)
• Dependency injection via python-dotenv & Flask-CORS
d. Key technologies & frameworks
• Flask (web framework)
• Flask-CORS (cross-origin support)
• requests (HTTP client)
• pandas + openpyxl (data processing & Excel export)
• python-dotenv (configuration management)
• Docker + docker-compose (containerization & orchestration)
– Frontend (e.g., blue)
– Backend services (green)
– External APIs (orange)
– Infrastructure (gray for Docker/volume)
• Annotations for protocols (HTTP/HTTPS), ports (5000), and configuration sources (.env)