# shaz-cy/hybridlogiq-smart-log-anomaly-explorer architecture

> Architecture diagram of the GitHub repository shaz-cy/hybridlogiq-smart-log-anomaly-explorer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer
- Repository: https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer

## Overview

##### Project Type & Purpose
- Full-stack, end-to-end anomaly detection platform.
- Backend: Python/Flask API performing log ingestion, NLP embedding (RoBERTa), autoencoder scoring, SOM clustering.
- Frontend: React.js + Tailwind CSS dashboard for uploading logs and visualizing results in real time.

##### File Structure
- Top level:
- AnomalyBackend/ – Flask app, data pipelines, model artifacts, utilities
- anomaly-dashboard/ – React dashboard code, components, chart libraries
- README.md, LICENSE
- AnomalyBackend subdirectories:
- models/ – trained PyTorch autoencoders, scalers, SOM pickles, metadata
- training_data/, uploads/, results/ – raw logs, generated training logs, anomaly output JSON
- Frontend:
- src/components/ – uploader, charts, tables, status & insights views
- src/components/charts/ – reusable chart components
- public/index.html, tailwind.config.js, package.json

##### Key Architecture & Patterns
- Monolithic Flask service exposing REST endpoints (e.g., POST /upload, GET /results).
- Pipeline pattern in backend: FileHandler → Embedder (RoBERTa) → Autoencoder → SOM → Results writer.
- React component hierarchy for UI: top-level App → pages (FileUploader, TrainingUploader) → visualization components.

##### Main Components
a. Frontend (React Dashboard)
- Browser UI
- Components: FileUploader, TrainingUploader, Visualizations (charts, tables)
- Data flow: HTTP(S) requests to Flask API
b. API Gateway / Flask Backend
- Endpoints:
- /upload/logs → FileHandler parses, writes to AnomalyBackend/uploads
- /upload/training → triggers clean_training_data & model re-training (optional)
- Modules:
- FileHandler (file_handler.py)
- NLP Embedder (HuggingFace RoBERTa)
- Autoencoder (PyTorch)
- Scaler (scikit-learn)
- SOM (MiniSom)
- Visualizer (generates JSON results in AnomalyBackend/results)
c. Data Stores & Artifacts
- File system directories: uploads/, training_data/, results/
- models/: persisted .pt, .save, .pkl files + metadata
d. External Dependencies
- HuggingFace Transformers API
- PyTorch compute environment (CPU/GPU)
- scikit-learn library
- Node/NPM for frontend dev server

##### Relationships & Data Flow
1. User uploads log file via React → HTTP POST to Flask
2. Flask FileHandler saves raw logs → triggers pipeline
3. Embedder module transforms logs into vector embeddings
4. Scaler normalizes features → passes to autoencoder
5. Autoencoder computes reconstruction error = anomaly score
6. SOM clusters anomalies → assigns BMU and cluster IDs
7. Visualizer converts outputs to JSON (anomalies.json, result_*.json)
8. Frontend polls or requests GET /results → renders charts & tables

##### Architectural Patterns & Principles
- Separation of concerns: distinct modules for I/O, embedding, scoring, clustering, visualization
- Pipeline/chain-of-responsibility for data processing
- Stateless API calls; state persisted as files
- Component-based UI with reusable chart-building blocks

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Layer
    subgraph "Frontend (React Dashboard)"
        direction TB
        FU["FileUploader Component"]:::frontend
        TU["TrainingUploader Component"]:::frontend
        VisCont["Visualizations Container"]:::frontend
        subgraph "Visualization Components"
            direction TB
            RecChart["Reconstruction Error Chart"]:::frontend
            RawLogView["Raw Log View"]:::frontend
            ResultTable["Result Table"]:::frontend
            SeverityTable["Severity Table"]:::frontend
            SomClusterTable["SOM Cluster Table"]:::frontend
            SummaryCards["Summary Cards"]:::frontend
            SystemStatus["System Status Panel"]:::frontend
            ModelInsights["Model Insights View"]:::frontend
            BMUMap["BMU Map Chart"]:::frontend
            GridHeatmap["Grid Search Heatmap Chart"]:::frontend
            HeatmapChart["Heatmap Chart"]:::frontend
            HistogramChart["Histogram Chart"]:::frontend
            LineChart["Line Chart"]:::frontend
            SomScatterChart["SOM Scatter Chart"]:::frontend
        end
    end

    %% Backend Layer
    subgraph "Backend (Flask API)"
        direction TB
        API["API Gateway / Flask App"]:::backend
        FH["File Handler Module"]:::backend
        Analyzer["RoBERTa Embedder"]:::backend
        Autoencoder["Autoencoder Scoring Module"]:::backend
        SOM["SOM Clustering Module"]:::backend
        Visualizer["Visualization Generator"]:::backend
        CleanLogs["Clean Uploaded Logs Script"]:::backend
        CleanTrain["Clean Training Data & Retrain Script"]:::backend
    end

    %% Storage Layer
    subgraph "Storage (File System)"
        direction TB
        Uploads["Raw Uploads Directory"]:::storage
        Training["Training Data Directory"]:::storage
        Results["Anomaly Results Directory"]:::storage
        Models["Model Artifacts Directory"]:::storage
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        HFA["HuggingFace Transformers API"]:::external
        PyTorch["PyTorch Compute Env"]:::external
        Sklearn["scikit-learn Library"]:::external
        Node["Node/NPM Dev Server"]:::external
    end

    %% Data Flow
    FU -->|"1 POST /upload/logs"| API
    TU -->|"1 POST /upload/training"| API
    API -->|"2 save logs"| FH
    FH -->|"3 generate embeddings"| Analyzer
    Analyzer -->|"4 scale & encode"| Autoencoder
    Autoencoder -->|"5 cluster"| SOM
    SOM -->|"6 convert to JSON"| Visualizer
    Visualizer -->|"7 write results"| Results
    API -->|"8 GET /results"| VisCont
    VisCont --> RecChart
    VisCont --> RawLogView
    VisCont --> ResultTable
    VisCont --> SeverityTable
    VisCont --> SomClusterTable
    VisCont --> SummaryCards
    VisCont --> SystemStatus
    VisCont --> ModelInsights
    VisCont --> BMUMap
    VisCont --> GridHeatmap
    VisCont --> HeatmapChart
    VisCont --> HistogramChart
    VisCont --> LineChart
    VisCont --> SomScatterChart

    %% External Integration
    Analyzer --> HFA
    Autoencoder --> PyTorch
    Autoencoder --> Sklearn
    FU --> Node
    TU --> Node

    %% Click Events
    click FU "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/FileUploader.jsx"
    click TU "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/TrainingUploader.jsx"
    click VisCont "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/Visualizations.jsx"
    click RecChart "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/ReconstructionChart.jsx"
    click RawLogView "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/RawLogView.jsx"
    click ResultTable "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/ResultTable.jsx"
    click SeverityTable "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/SeverityTable.jsx"
    click SomClusterTable "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/SomClusterTable.jsx"
    click SummaryCards "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/SummaryCards.jsx"
    click SystemStatus "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/SystemStatus.jsx"
    click ModelInsights "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/ModelInsights.jsx"
    click BMUMap "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/charts/BmuMap.jsx"
    click GridHeatmap "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/charts/GridSearchHeatmap.jsx"
    click HeatmapChart "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/charts/HeatmapChart.jsx"
    click HistogramChart "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/charts/HistogramChart.jsx"
    click LineChart "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/charts/LineChart.jsx"
    click SomScatterChart "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/anomaly-dashboard/src/components/charts/SomScatterChart.jsx"
    click API "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/AnomalyBackend/app.py"
    click FH "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/AnomalyBackend/utils/file_handler.py"
    click Analyzer "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/AnomalyBackend/utils/analyzer.py"
    click Autoencoder "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/AnomalyBackend/utils/autoencoder.py"
    click Visualizer "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/AnomalyBackend/utils/visualize.py"
    click CleanLogs "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/AnomalyBackend/clean_logs.py"
    click CleanTrain "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/blob/main/AnomalyBackend/clean_training_data.py"
    click Uploads "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/tree/main/AnomalyBackend/uploads/"
    click Training "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/tree/main/AnomalyBackend/training_data/"
    click Results "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/tree/main/AnomalyBackend/results/"
    click Models "https://github.com/shaz-cy/hybridlogiq-smart-log-anomaly-explorer/tree/main/AnomalyBackend/models/"

    %% Styles
    classDef frontend fill:#e6ffed,stroke:#2d7a2d,color:#064e18
    classDef backend fill:#e6f0ff,stroke:#3465a4,color:#1d2d50
    classDef storage fill:#f0f0f0,stroke:#888888,color:#333333
    classDef external fill:#fff4e6,stroke:#e08e0b,color:#854200
```
