# aluminumpirate/py-threat-intelligence architecture

> Architecture diagram of the GitHub repository aluminumpirate/py-threat-intelligence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aluminumpirate/py-threat-intelligence
- Repository: https://github.com/aluminumpirate/py-threat-intelligence

## Overview

##### Project type and purpose
- This is a full‑stack, containerized Threat Intelligence System composed of:
- A Python FastAPI‑based REST API server (app/)
- A React/Vite frontend (frontend/)
- A cron‑based scheduler service (scheduler/)
- A PostgreSQL database (configured via docker‑compose)
- Purpose: collect, store, and present domain threat data via multiple OSINT services (VirusTotal, WHOIS, DNS providers).

##### File structure
- app/
- main.py – entrypoint for FastAPI
- controllers/ – HTTP handlers mapping to routes
- routers/ – FastAPI router definitions (domains, scans, jobs)
- services/ – business logic (scan orchestration, service wrappers)
- models.py, schemas.py – SQLAlchemy ORM models & Pydantic schemas
- database.py – async engine & session setup
- frontend/
- src/ – React components, pages, API client (api.js)
- index.html, package.json, vite.config.js – frontend build config
- scheduler/
- scan_job.py – script to run scheduled scans
- crontab.txt – cron expression
- docker-compose.yml orchestrates API, scheduler, database, and frontend
- requirements.txt & scheduler/requirements‑scheduler.txt list Python dependencies

##### README for additional insights
- Tech stack: Python 3.8+, FastAPI, SQLAlchemy, Pydantic, React (Vite), Postgres, Docker, cron
- Three components: API server, database, scheduler
- Docker‑compose for local deployment; environment variables for keys & DB URL

## Diagram (Mermaid)

```mermaid
graph TB
    %% Docker Compose Boundary
    subgraph "Docker Compose" 
        direction TB

        %% Frontend Container
        subgraph "Frontend Container"
            direction TB
            Frontend["Frontend\n(React/Vite)"]:::frontend
        end

        %% API Server Container
        subgraph "API Server Container"
            direction TB
            Main["main.py"]:::backend
            RoutersSub["Routers"]:::backend
            ControllersSub["Controllers"]:::backend
            ServicesSub["Services"]:::backend
            ModelsSub["Models/Schemas/DB"]:::backend
        end

        %% Scheduler Container
        subgraph "Scheduler Container"
            direction TB
            ScanJob["scan_job.py"]:::backend
            Crontab["crontab.txt"]:::backend
        end

        %% Database Container
        subgraph "PostgreSQL Container"
            direction TB
            PostgresDB["PostgreSQL DB"]:::database
        end
    end

    %% External OSINT Services
    subgraph "External OSINT Services"
        direction TB
        VirusTotal["VirusTotal API"]:::external
        WHOIS["WHOIS API"]:::external
        Cloudflare["Cloudflare DNS"]:::external
        GoogleDNS["Google DNS"]:::external
        HackerTarget["HackerTarget DNS"]:::external
    end

    %% API internal structure expansions
    subgraph "API Server Internal" 
        direction TB
        RoutersSub --> DomainRouter["domain.py"]:::backend
        RoutersSub --> DomainsRouter["domains.py"]:::backend
        RoutersSub --> JobsRouter["jobs.py"]:::backend

        ControllersSub --> DomainController["domain_controller.py"]:::backend
        ControllersSub --> ScanController["scan_controller.py"]:::backend

        ServicesSub --> ScanService["scan_service.py"]:::backend
        ServicesSub --> BaseService["base_service.py"]:::backend
        ServicesSub --> VirusTotalService["virus_total_service.py"]:::backend
        ServicesSub --> WhoisService["whois_service.py"]:::backend
        ServicesSub --> CloudflareDNSService["cloudflare_dns_service.py"]:::backend
        ServicesSub --> GoogleDNSService["google_dns_service.py"]:::backend
        ServicesSub --> HackerTargetDNSService["hackertarget_dns_service.py"]:::backend

        ModelsSub --> ModelsFile["models.py"]:::backend
        ModelsSub --> SchemasFile["schemas.py"]:::backend
        ModelsSub --> DatabaseSetup["database.py"]:::backend
    end

    %% Connections
    Frontend -->|"REST API (GET/POST)"| Main
    Main --> RoutersSub
    DomainRouter --> DomainController
    DomainsRouter --> DomainController
    JobsRouter --> ScanController
    DomainController --> ScanService
    ScanController --> ScanService
    ScanService --> PostgresDB
    DatabaseSetup --> PostgresDB

    %% External OSINT calls
    ScanService --> VirusTotal
    ScanService --> WHOIS
    ScanService --> Cloudflare
    ScanService --> GoogleDNS
    ScanService --> HackerTarget

    %% Scheduler interactions
    ScanJob --> ScanService
    ScanJob --> PostgresDB
    Crontab --> ScanJob

    %% Docker-Compose orchestration note
    Compose["docker-compose.yml"]:::config
    Compose --> Frontend
    Compose --> Main
    Compose --> ScanJob
    Compose --> PostgresDB

    %% Click Events
    click Frontend "https://github.com/aluminumpirate/py-threat-intelligence/tree/main/frontend/"
    click Main "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/main.py"
    click DomainRouter "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/routers/domain.py"
    click DomainsRouter "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/routers/domains.py"
    click JobsRouter "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/routers/jobs.py"
    click DomainController "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/controllers/domain_controller.py"
    click ScanController "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/controllers/scan_controller.py"
    click ScanService "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/services/scan_service.py"
    click BaseService "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/services/base_service.py"
    click VirusTotalService "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/services/virus_total_service.py"
    click WhoisService "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/services/whois_service.py"
    click CloudflareDNSService "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/services/cloudflare_dns_service.py"
    click GoogleDNSService "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/services/google_dns_service.py"
    click HackerTargetDNSService "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/services/hackertarget_dns_service.py"
    click ModelsFile "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/models.py"
    click SchemasFile "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/schemas.py"
    click DatabaseSetup "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/app/database.py"
    click ScanJob "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/scheduler/scan_job.py"
    click Crontab "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/scheduler/crontab.txt"
    click Compose "https://github.com/aluminumpirate/py-threat-intelligence/blob/main/docker-compose.yml"

    %% Styles
    classDef frontend fill:#AEDFF7,stroke:#039BE5,color:#000
    classDef backend fill:#D5F5E3,stroke:#27AE60,color:#000
    classDef database fill:#FDEBD0,stroke:#D35400,color:#000
    classDef external fill:#ECECEC,stroke:#A6ACAF,color:#000
    classDef config fill:#F0F3F4,stroke:#7B7D7D,color:#000
```
