# paladin316/byo-secai architecture

> Architecture diagram of the GitHub repository paladin316/byo-secai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paladin316/byo-secai
- Repository: https://github.com/paladin316/byo-secai

## Overview

##### Project type and purpose
- This is a local-first AI assistant for cybersecurity research. It orchestrates a large language model (Ollama) and a web UI (OpenWebUI/Streamlit) plus custom “plugin” scripts for threat hunting, detection, and incident response. It also offers a Retrieval-Augmented Generation (RAG) ingest/query pipeline.

##### File structure
- Top level:
- Batch launchers (.bat) for different modes (CLI, Streamlit UI, RAG)
- config.py / config.yaml – central configuration
- data/ – sample or user data for ingestion
- docs/, CONTRIBUTING.md, LICENSE – project docs
- logs/ – runtime logs (e.g., ingestion_log.txt)
- plugin_loader.py + plugins/ – dynamic plugin architecture
- scripts/ – orchestration scripts (LLM orchestrator, RAG ingest/query, UI runner, installation helpers)

##### README for additional insights
- Stacks: Python 3.10, Docker, Ollama (phi model), OpenWebUI, Streamlit
- Purpose: Local security assistant with plugin-driven threat-intel lookups and RAG

##### Main components & their interactions
a. User Interfaces
– CLI (Launch_CLI_Assistant.bat → python scripts/llm_orchestrator_with_plugins.py)
– Streamlit RAG UI (Launch_Streamlit_UI.bat → scripts/streamlit_rag_ui.py)
– OpenWebUI (Docker container connecting to Ollama endpoint)
b. Core Orchestrator
– plugin_loader.py loads plugin modules (abuseipdb.py, urlscan.py, virustotal.py, yara_scanner.py)
– scripts/llm_orchestrator_with_plugins.py routes user prompts → plugins → LLM
c. LLM Host
– Ollama (phi model) running in Docker, exposed on localhost:11434
– OpenWebUI connects to that host for visual interface
d. RAG Pipeline
– scripts/rag_ingest_and_query.py builds vector index from data/ and docs/
– scripts/rag_safety_checklist.py supplies audit checks
– Query flow: UI or CLI → RAG query script → vector store → LLM
e. Data & Configuration
– config.yaml / config.py – define model endpoint, plugin settings
– data/ – source docs, example.txt
– logs/ – ingestion_log.txt, other logs
f. External Services
– AbuseIPDB, URLScan.io, VirusTotal (via plugins)

##### Architectural patterns & design principles
- Plugin architecture: plugin_loader.py dynamically imports threat-intel connectors
- Pipeline pattern: separation of ingest (index creation) vs. query
- Local-first, container-backed LLM hosting
- Single‐binary “orchestrator” microservice (Python) with distinct UI runners

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Launchers
    LAUNCH_CLI["Launch_CLI_Assistant.bat"]:::frontend
    LAUNCH_STREAM["Launch_Streamlit_UI.bat"]:::frontend
    LAUNCH_RAG["Launch_LLM-Powered_RAG_Assistant.bat"]:::frontend

    %% User Interfaces
    subgraph "User Interfaces"
        CLI["CLI Client"]:::frontend
        STREAM["Streamlit RAG UI"]:::frontend
        OPENWEB["OpenWebUI"]:::frontend
    end

    %% Core Orchestrator
    subgraph "Orchestrator Layer"
        ORCH["Orchestrator (Python)"]:::orchestrator
    end

    %% Plugin Modules
    subgraph "Plugins"
        AIPDB["AbuseIPDB Plugin"]:::plugins
        URLSCAN["URLScan.io Plugin"]:::plugins
        VT["VirusTotal Plugin"]:::plugins
        YARA["Yara Scanner Plugin"]:::plugins
    end

    %% RAG Pipeline
    subgraph "RAG Pipeline"
        INGEST["RAG Ingest"]:::orchestrator
        SAFETY["RAG Safety Checklist"]:::orchestrator
        VECTOR["Vector Store"]:::data
    end

    %% Model Host
    subgraph "Model Host"
        OLLAMA["Ollama φ (Docker)"]:::llm
    end

    %% External Services
    subgraph "External Services"
        ABUSE["AbuseIPDB API"]:::external
        URLSVC["URLScan.io API"]:::external
        VTSVC["VirusTotal API"]:::external
    end

    %% Configuration & Data
    subgraph "Configuration & Data"
        CFGPY["config.py"]:::data
        CFGYAML["config.yaml"]:::data
        DATADIR["data/"]:::data
        LOGSDIR["logs/"]:::data
    end

    %% Connections: Launchers to UIs/Pipeline
    LAUNCH_CLI --> CLI
    LAUNCH_STREAM --> STREAM
    LAUNCH_RAG --> INGEST

    %% UI to Orchestrator/Pipeline
    CLI --> ORCH
    STREAM --> INGEST
    OPENWEB --> OLLAMA

    %% Orchestrator to LLM and Plugins
    ORCH -->|HTTP calls| OLLAMA
    ORCH --> AIPDB
    ORCH --> URLSCAN
    ORCH --> VT
    ORCH --> YARA

    %% Plugins to External Services
    AIPDB --> ABUSE
    URLSCAN --> URLSVC
    VT --> VTSVC

    %% RAG Pipeline flow
    INGEST --> VECTOR
    SAFETY --> VECTOR
    STREAM --> VECTOR
    VECTOR --> ORCH

    %% Config & Data usage
    CFGPY --> ORCH
    CFGYAML --> ORCH
    DATADIR --> INGEST
    DATADIR --> VECTOR
    LOGSDIR --> INGEST

    %% Click Events
    click LAUNCH_CLI "https://github.com/paladin316/byo-secai/blob/main/Launch_CLI_Assistant.bat"
    click LAUNCH_STREAM "https://github.com/paladin316/byo-secai/blob/main/Launch_Streamlit_UI.bat"
    click LAUNCH_RAG "https://github.com/paladin316/byo-secai/blob/main/Launch_LLM-Powered_RAG_Assistant.bat"
    click CLI "https://github.com/paladin316/byo-secai/blob/main/Launch_CLI_Assistant.bat"
    click STREAM "https://github.com/paladin316/byo-secai/blob/main/Launch_Streamlit_UI.bat"
    click ORCH "https://github.com/paladin316/byo-secai/blob/main/scripts/llm_orchestrator_with_plugins.py"
    click AIPDB "https://github.com/paladin316/byo-secai/blob/main/plugins/abuseipdb.py"
    click URLSCAN "https://github.com/paladin316/byo-secai/blob/main/plugins/urlscan.py"
    click VT "https://github.com/paladin316/byo-secai/blob/main/plugins/virustotal.py"
    click YARA "https://github.com/paladin316/byo-secai/blob/main/plugins/yara_scanner.py"
    click SAFETY "https://github.com/paladin316/byo-secai/blob/main/scripts/rag_safety_checklist.py"
    click INGEST "https://github.com/paladin316/byo-secai/blob/main/scripts/rag_ingest_and_query.py"
    click DATADIR "https://github.com/paladin316/byo-secai/tree/main/data/"
    click LOGSDIR "https://github.com/paladin316/byo-secai/blob/main/logs/ingestion_log.txt"
    click CFGPY "https://github.com/paladin316/byo-secai/blob/main/config.py"
    click CFGYAML "https://github.com/paladin316/byo-secai/blob/main/config.yaml"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085
    classDef orchestrator fill:#d4edda,stroke:#155724,color:#155724
    classDef llm fill:#fff3cd,stroke:#856404,color:#856404
    classDef plugins fill:#e2d6f3,stroke:#5e3d9a,color:#5e3d9a
    classDef data fill:#f8f9fa,stroke:#6c757d,color:#6c757d
    classDef external fill:#ffe5e5,stroke:#a71d2a,color:#a71d2a
```
