# martel-innovate/horse-genai-ckb architecture

> Architecture diagram of the GitHub repository martel-innovate/horse-genai-ckb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/martel-innovate/horse-genai-ckb
- Repository: https://github.com/martel-innovate/horse-genai-ckb

## Overview

##### Major Components
- Centralized Database: Represented by the PostgreSQL database, which is set up using Docker Compose (with configuration files such as .env.example, docker-compose.yml, and the init.sql script). This component stores the attack data and enriched mitigation information.
- Data Acquisition & Scraping (Dynamic Attack Resolver): This includes the WebScraper modules (see src/scrape_and_generate.py, src/scrape_attacks.py, src/scrape_mitigations.py, src/scraper.py) that extract attack patterns and mitigation strategies from CAPEC.
- ETL Process: The component (src/extract_transform_load.py) that converts the scraped and AI-generated data into a structured format, making it ready for integration into the central database.
- Mitigation Generator: Comprising the mitigation_generator.py and generate_from_csv.py modules, this component uses generative AI (via LLMs such as Llama 3.1, Claude 3.5 Sonnet, DeepSeek R1 8B, Granite 3.0 8B, Llama 2 13B/7B, and ALIENTELLIGENCE-cyberaisecurity) to generate and enhance mitigation responses.
- Evaluation Module: Although not a runtime component, the evaluation and notebook directories (evaluation/LLMs and evaluation/notebooks) provide CSV results and Jupyter visualizations detailing the performance of various models.
- External Services & Integrations: Incorporate the connection to external LLM providers (Ollama for local model serving and proprietary APIs for models like Claude) and the RESTful API service mentioned in the README for external communication.

##### Relationships and Data Flow
- Data originates at the CAPEC website and flows into the WebScraper modules, which output raw data (stored into CSV files found in the ‘output’ folder).
- This scraped data is then processed by the ETL script to format it appropriately.
- The formatted data is sent to the Mitigation Generator, which invokes external generative AI models. The generated mitigation responses are saved back into the ‘output’ folder (each file named by model) for traceability.
- The Centralized Database then receives the processed data (as handled via ETL) to maintain the enhanced Common Knowledge Base.
- The Evaluation component separately processes the outputs from various LLMs, providing performance assessments that inform the accuracy and quality of the mitigations.

##### Architectural Patterns and Technologies
- Modular Design: The project cleanly separates functions into components (Scraping, ETL, AI Generation, and Evaluation).
- Pipeline/Workflow Pattern: Data flows sequentially from scraping to generation, evaluation, and finally into the database.
- Containerization: The use of Docker Compose emphasizes containerized deployment for the PostgreSQL database.
- Integration with External APIs: Interaction with various generative AI models through both local (Ollama for Llama models) and remote (Claude) APIs.
- Extensibility: The modular approach allows you to plug in additional models or modify components (like swapping ETL processes or adding new evaluation criteria) without affecting the entire system.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Source
    CAPEC(("CAPEC Website")):::external

    %% Data Acquisition & Scraping
    WebScraper["CAPEC WebScraper"]:::scraping

    %% ETL Process
    ETL["ETL Pipeline"]:::etl

    %% Mitigation Generation
    MitGen["Mitigation Generator"]:::mitigation

    %% Mitigation Outputs
    MitOutputs["Mitigation Outputs (CSV)"]:::mitigation

    %% Centralized Database
    DB["Centralized Database (PostgreSQL)"]:::database

    %% External LLM API Integration
    ExternalLLM(("External LLM API Service")):::external

    %% Evaluation Module
    Evaluation["LLM Evaluation Module"]:::evaluation

    %% Data Flow
    CAPEC -->|"scrapes"| WebScraper
    WebScraper -->|"extracts_data"| ETL
    ETL -->|"transforms"| MitGen
    MitGen -->|"stores_to"| DB
    MitGen -->|"outputs"| MitOutputs
    MitOutputs -->|"evaluated_by"| Evaluation
    Evaluation -->|"feedback"| MitGen
    MitGen -- "calls" --> ExternalLLM
    ExternalLLM -- "returns" --> MitGen

    %% Click Events for Centralized Database
    click DB "https://github.com/martel-innovate/horse-genai-ckb/blob/main/.env.example"
    click DB "https://github.com/martel-innovate/horse-genai-ckb/blob/main/docker-compose.yml"
    click DB "https://github.com/martel-innovate/horse-genai-ckb/blob/main/init.sql"

    %% Click Events for Data Acquisition & Scraping
    click WebScraper "https://github.com/martel-innovate/horse-genai-ckb/blob/main/src/scrape_and_generate.py"
    click WebScraper "https://github.com/martel-innovate/horse-genai-ckb/blob/main/src/scrape_attacks.py"
    click WebScraper "https://github.com/martel-innovate/horse-genai-ckb/blob/main/src/scrape_mitigations.py"
    click WebScraper "https://github.com/martel-innovate/horse-genai-ckb/blob/main/src/scraper.py"
    click WebScraper "https://github.com/martel-innovate/horse-genai-ckb/tree/main/output/scraper"

    %% Click Event for ETL Process
    click ETL "https://github.com/martel-innovate/horse-genai-ckb/blob/main/src/extract_transform_load.py"

    %% Click Events for Mitigation Generator
    click MitGen "https://github.com/martel-innovate/horse-genai-ckb/blob/main/src/mitigation_generator.py"
    click MitGen "https://github.com/martel-innovate/horse-genai-ckb/blob/main/src/generate_from_csv.py"

    %% Click Event for Mitigation Outputs
    click MitOutputs "https://github.com/martel-innovate/horse-genai-ckb/tree/main/output"

    %% Click Events for Evaluation Module
    click Evaluation "https://github.com/martel-innovate/horse-genai-ckb/tree/main/evaluation/LLMs"
    click Evaluation "https://github.com/martel-innovate/horse-genai-ckb/tree/main/evaluation/notebooks"

    %% Classes and Styling
    classDef external fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef scraping fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef etl fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef mitigation fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef database fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef evaluation fill:#f8d7da,stroke:#721c24,stroke-width:2px;
```
