# centerforcyberintelligence/threatinsight-analyzer architecture

> Architecture diagram of the GitHub repository centerforcyberintelligence/threatinsight-analyzer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/centerforcyberintelligence/threatinsight-analyzer
- Repository: https://github.com/centerforcyberintelligence/threatinsight-analyzer

## Overview

##### Main Components
- Client Interface:
- Web-based frontend built with HTML/CSS, JavaScript, and HTMX.
- Separate templates, static assets, and partial views (e.g., index.html, statistics.html, and various partials).
- Backend:
- The Flask Application serving as the primary request handler.
- Application factory (app/__init__.py) initializing configurations and blueprints.
- Multiple route blueprints (under app/blueprints/) for functionalities such as analysis, history, settings, and statistics.
- AI Processing:
- Utility modules (in app/utilities/) like article_analyzer.py, article_extractor.py, and indicator_extractor.py. These modules interact with external AI services via the OpenAI API, implementing structured JSON schema validation and error-handling.
- Database:
- SQLite database managed via modules in app/models/database.py. It stores analysis results, history, and includes migration functionality.
- Configuration & Environment:
- Environment variable management (provided via .env.template) and configuration modules (app/config/config.py) which supply settings for API keys, Flask server settings, and other variables.
- Additional Tools:
- Logging (app/utilities/logger.py) and export utilities (app/utilities/export.py) for reporting and diagnostics.
- CLI or migration scripts (scripts/migrate_data.py) for data/database management.

##### Relationships and Interactions
- The Client Interface (browser/Streamlit) sends HTTP requests which route through the Flask application.
- The Flask blueprints separate the core functions:
- The Analysis blueprint interfaces with the AI utilities to process and analyze articles.
- History and statistics blueprints manage and display analytical data.
- The utility components interact with external systems:
- Article extraction and sanitization are applied before sending data to the AI service.
- API calls to the OpenAI endpoint (using parameters set in configuration) return structured responses.
- Database operations occur where insights and results are written to and read from SQLite tables.
- There is also an export flow from the utilities to generate output in various formats (JSON, CSV, etc.).

##### Architectural Patterns and Design Principles
- Modular design through separation of concerns via Flask blueprints.
- Utilizing utility libraries to encapsulate support functions (article extraction, sanitization, etc.).
- Clear application layering: presentation (templates/static), business logic (blueprints, utilities), and data persistence (models/database).
- Emphasis on API integration with structured JSON responses and error-handling patterns (retry, exponential backoff).

##### Relevant Technologies, Frameworks, and Libraries
- Python as the core language.
- Flask as the web framework.
- HTMX for dynamic frontend components.
- SQLite for the database.
- OpenAI API for AI processing.
- Newspaper3k (for content extraction) and regex-based methods for IOC extraction.
- Testing using pytest to ensure reliability.

##### Specific Elements
- Clear labels for each component (e.g., “Frontend – HTML/HTMX”, “Flask Application Factory”, “Analysis Blueprint”, “Utility Modules: Article Analyzer/Extractor”, “SQLite Database”).
- From the Web Client to Blueprints.
- From Blueprints to Utility Modules.
- From Utility Modules to the OpenAI API.
- From Blueprints to the Database.
- Presentation Layer (Client Interface, Templates, Static assets)
- Application Layer (Flask Blueprints, Configuration)
- Data Processing & Integration Layer (Utilities, AI API)
- Persistence Layer (Database models and actual SQLite instance)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        CI["Client Interface: Web Browser (HTML/HTMX) & Streamlit"]:::frontend
        TPL["Templates"]:::frontend
        STATIC["Static Assets"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Layer"
        RP["run.py (Main Entry Point)"]:::backend
        FAF["Flask Application Factory"]:::backend
        CFG["Configuration Module"]:::backend
        ENV["Environment Template"]:::backend
        subgraph "Flask Blueprints"
            AB["Analysis Blueprint"]:::blueprint
            HB["History Blueprint"]:::blueprint
            SB["Settings Blueprint"]:::blueprint
            STB["Statistics Blueprint"]:::blueprint
            MB["Main Blueprint"]:::blueprint
        end
    end

    %% Data Processing & AI Integration
    subgraph "Data Processing & AI Integration"
        AA["Article Analyzer"]:::utilities
        AE["Article Extractor"]:::utilities
        IE["Indicator Extractor"]:::utilities
        EU["Export Utilities"]:::utilities
        LG["Logging"]:::utilities
        OA["OpenAI API"]:::external
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        DB["SQLite Database"]:::db
        MIG["CLI/Migration Script"]:::backend
    end

    %% Connections between layers
    CI -->|"HTTP_Request"| RP
    TPL --> CI
    STATIC --> CI

    RP -->|"Initializes"| FAF
    FAF -->|"Loads_Config"| CFG
    CFG -->|"Uses"| ENV

    FAF -->|"Registers"| AB
    FAF -->|"Registers"| HB
    FAF -->|"Registers"| SB
    FAF -->|"Registers"| STB
    FAF -->|"Registers"| MB

    %% Blueprints interacting with AI Processing and DB
    AB -->|"Calls_Analyzer"| AA
    AB -->|"Calls_Extractor"| AE
    AB -->|"Calls_Indicator"| IE
    AB -->|"Exports_Data"| EU
    AB -->|"Logs"| LG
    HB -->|"Accesses_DB"| DB
    STB -->|"Accesses_DB"| DB
    MB -->|"DefaultRoutes_DB"| DB

    %% AI Utilities interacting with External API
    AA -->|"Uses_API"| OA
    AE -->|"Uses_API"| OA
    IE -->|"Uses_API"| OA

    %% Backend DB operations
    FAF -->|"DB_Operations"| DB
    MIG -->|"Migrates_DB"| DB

    %% Click Events
    click TPL "https://github.com/centerforcyberintelligence/threatinsight-analyzer/tree/main/templates/"
    click STATIC "https://github.com/centerforcyberintelligence/threatinsight-analyzer/tree/main/static/"
    click RP "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/run.py"
    click FAF "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/__init__.py"
    click CFG "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/config/config.py"
    click ENV "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/.env.template"
    click AB "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/blueprints/analysis.py"
    click HB "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/blueprints/history.py"
    click SB "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/blueprints/settings.py"
    click STB "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/blueprints/statistics.py"
    click MB "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/blueprints/main.py"
    click AA "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/utilities/article_analyzer.py"
    click AE "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/utilities/article_extractor.py"
    click IE "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/utilities/indicator_extractor.py"
    click LG "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/utilities/logger.py"
    click EU "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/utilities/export.py"
    click DB "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/app/models/database.py"
    click MIG "https://github.com/centerforcyberintelligence/threatinsight-analyzer/blob/main/scripts/migrate_data.py"

    %% Styles
    classDef frontend fill:#d9edf7,stroke:#31708f,stroke-width:2px;
    classDef backend fill:#dff0d8,stroke:#3c763d,stroke-width:2px;
    classDef blueprint fill:#fcf8e3,stroke:#8a6d3b,stroke-width:2px;
    classDef utilities fill:#d9edf7,stroke:#31708f,stroke-width:2px;
    classDef db fill:#f2dede,stroke:#a94442,stroke-width:2px;
    classDef external fill:#eeeeee,stroke:#666666,stroke-width:2px;
```
