# mhmdsan7/ioc_gatherer architecture

> Architecture diagram of the GitHub repository mhmdsan7/ioc_gatherer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mhmdsan7/ioc_gatherer
- Repository: https://github.com/mhmdsan7/ioc_gatherer

## Overview

##### Project type and purpose
- This is a full-stack, web-based threat-intelligence tool built with Flask.
- Purpose: gather Indicators of Compromise (IoCs) from multiple external feeds, display results in the browser, and export reports to Excel.

##### File structure
- app.py: the main Flask application, defines UI routes and REST endpoints (`/api/search`, `/api/export`).
- templates/index.html: single-page web UI.
- requirements.txt: lists Python dependencies (Flask, Flask-CORS, requests, pandas, openpyxl, python-dotenv).
- .env.template: placeholder for API keys (AbuseIPDB, VirusTotal, ThreatFox, NIST NVD) loaded via python-dotenv.
- Dockerfile + docker-compose.yaml: containerize the Flask app and expose port 5000; mount a volume for exported files; manage container lifecycle and health checks.

##### README for additional insights
- Multi-feed support, IoC analysis across four types (IP, domain, hash, CVE).
- External intelligence sources accessed via HTTP.
- Excel export: pandas + openpyxl produce formatted workbooks.
- Two deployment modes: local venv and Docker (with compose orchestration).

a. Main components (container-level view)
1. User Browser (React-style single page? Just HTML+JS in index.html)
2. Flask Web Server Container
- HTTP Server & Routing
- UI Controller (renders index.html)
- API Layer
– Search Service (aggregates IoCs across feeds)
– Export Service (formats data into Excel)
- Service Clients for each external API:
– AbuseIPDB Client
– VirusTotal Client
– ThreatFox Client
– NVD Client
- Data Processor (pandas pipeline)
- Configuration Loader (python-dotenv picks up .env keys)
3. Volume Storage (persistent volume for generated Excel files)
4. External Threat Intelligence APIs (4 services accessed over HTTPS)
5. Docker / Docker-Compose (hosts the Flask container, sets up env vars, health checks)

b. Relationships & data flows
- Browser → Flask UI Controller (HTTP GET) → index.html
- Browser → Flask API Layer (HTTP POST /api/search) →
- Passes feed definitions → Search Service → iterates over feeds → calls each Service Client → returns aggregated results → JSON response to browser
- Browser → Flask API Layer (HTTP POST /api/export) →
- Passes JSON results → Export Service → Data Processor (pandas → openpyxl) → writes to Volume Storage → returns download link or file stream
- Flask reads .env for API keys at startup
- Docker-Compose injects environment, maps port 5000, mounts storage volume

c. Architectural patterns & principles
- MVC-like separation: controllers (routes) vs. services (external API clients & data processing)
- Single-container micro-service, no internal database—stateless except for temp Excel files
- Configuration as code (.env, Docker-compose)
- Dependency injection via python-dotenv & Flask-CORS

d. Key technologies & frameworks
- Flask (web framework)
- Flask-CORS (cross-origin support)
- requests (HTTP client)
- pandas + openpyxl (data processing & Excel export)
- python-dotenv (configuration management)
- Docker + docker-compose (containerization & orchestration)

– Frontend (e.g., blue)
– Backend services (green)
– External APIs (orange)
– Infrastructure (gray for Docker/volume)
- Annotations for protocols (HTTP/HTTPS), ports (5000), and configuration sources (.env)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Layer
    subgraph "Frontend" 
        direction TB
        Browser["User Browser"]:::frontend
        Template["index.html (UI Template)"]:::frontend
    end

    %% Infrastructure Layer
    subgraph "Docker-Compose" 
        direction TB
        DockerCompose["Docker-Compose"]:::infra
        subgraph "Flask Container" 
            direction TB
            FlaskContainer["Flask Web Server"]:::backend
            ConfigLoader["Configuration Loader (.env)"]:::backend
            subgraph "Service Layer"
                direction TB
                SearchSvc["Search Service"]:::service
                ExportSvc["Export Service"]:::service
            end
            ServiceClients["External API Clients"]:::service
            DataProcessor["Data Processor (pandas)"]:::service
        end
        Volume["Volume Storage"]:::infra
    end

    %% External APIs
    subgraph "External Threat Intelligence APIs"
        direction TB
        AbuseIPDB["AbuseIPDB API"]:::external
        VirusTotal["VirusTotal API"]:::external
        ThreatFox["ThreatFox API"]:::external
        NVD["NVD API"]:::external
    end

    %% Documentation & Config Files
    Requirements["requirements.txt"]:::docs
    EnvTemplate[".env.template"]:::docs
    DockerfileNode["Dockerfile"]:::docs
    README["README.md"]:::docs

    %% Connections
    Browser -->|"loads"| Template
    Browser -->|HTTP GET /| FlaskContainer
    Browser -->|POST /api/search| SearchSvc
    SearchSvc -->|"calls"| ServiceClients
    ServiceClients -->|HTTPS| AbuseIPDB
    ServiceClients -->|HTTPS| VirusTotal
    ServiceClients -->|HTTPS| ThreatFox
    ServiceClients -->|HTTPS| NVD
    ServiceClients -->|results| SearchSvc
    SearchSvc -->|JSON response| Browser

    Browser -->|POST /api/export| ExportSvc
    ExportSvc -->|processes| DataProcessor
    DataProcessor -->|writes Excel| Volume
    Volume -->|download link| Browser

    ConfigLoader -->|loads keys| FlaskContainer
    DockerCompose -->|injects env & mounts| FlaskContainer
    DockerCompose -->|mounts| Volume

    %% File references
    click FlaskContainer "https://github.com/mhmdsan7/ioc_gatherer/blob/main/app.py"
    click Template "https://github.com/mhmdsan7/ioc_gatherer/blob/main/templates/index.html"
    click EnvTemplate "https://github.com/mhmdsan7/ioc_gatherer/blob/main/.env.template"
    click Requirements "https://github.com/mhmdsan7/ioc_gatherer/blob/main/requirements.txt"
    click DockerfileNode "https://github.com/mhmdsan7/ioc_gatherer/tree/main/Dockerfile"
    click DockerCompose "https://github.com/mhmdsan7/ioc_gatherer/blob/main/docker-compose.yaml"
    click README "https://github.com/mhmdsan7/ioc_gatherer/blob/main/README.md"

    %% Styling
    classDef frontend fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef backend fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef service fill:#98FB98,stroke:#333,stroke-width:1px;
    classDef external fill:#FFD580,stroke:#333,stroke-width:1px;
    classDef infra fill:#D3D3D3,stroke:#333,stroke-width:1px;
    classDef docs fill:#D8BFD8,stroke:#333,stroke-width:1px;
```
