# salesforce/logai architecture

> Architecture diagram of the GitHub repository salesforce/logai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/salesforce/logai
- Repository: https://github.com/salesforce/logai

## Overview

##### Main Components Identification
The system should be divided into these major components:

a. Core Components:
- Data Loading Layer
- Preprocessing Layer
- Information Extraction Layer
- Analysis Layer
- Applications Layer
- GUI Interface

b. Algorithm Components:
- Anomaly Detection Algorithms
- Clustering Algorithms
- Parsing Algorithms
- Vectorization Algorithms
- Neural Network Models

**┌─ GUI Layer ─────────────────────────────────────┐**
- Interactive Dashboard                         │
- Visualization Components                      │
- Configuration Interface                       │
- ┬───────────────────────────┘
- ↓
**┌─ Applications Layer ──────────────────────────┐**
- Log Anomaly Detection                       │
- Log Clustering                              │
- Log Summarization                           │
- ┬───────────────────────────┘
- ↓
**┌─ Analysis Layer ────────────────────────────────┐**
- Anomaly Detector                              │
- Clustering Engine                             │
- Neural Network Anomaly Detector               │
- ┬───────────────────────────┘
- ↓
**┌─ Information Extraction Layer ──────────────────┐**
- Log Parser                                    │
- Feature Extractor                            │
- Log Vectorizer                               │
- Categorical Encoder                          │
- ┬───────────────────────────┘
- ↓
**┌─ Preprocessing Layer ─────────────────────────┐**
- Data Partitioning                          │
- Data Cleaning                              │
- Format Standardization                     │
- ┬───────────────────────────┘
- ↓
**┌─ Data Loading Layer ──────────────────────────┐**
- OpenTelemetry Data Model                    │
- Custom Data Loaders                         │
- Data Model Conversion                       │
- ┬───────────────────────────┘
- ↓
**┌─ External Data Sources ────────────────────────┐**
- Log Files                                    │
- Streaming Data                               │
- OpenTelemetry Compatible Systems            │
- ┘

**┌─ Algorithm Ecosystem ────────────────────────────┐**

- ┌─ Anomaly Detection ─┐ ┌─ Clustering ────────┐ │
- One-Class SVM    │ │ - K-means           │ │
- Isolation Forest │ │ - DBSCAN            │ │
- LSTM            │ │ - BIRCH             │ │
- ┘ └──────────────────────┘ │

- ┌─ Parsing ─────────┐ ┌─ Vectorization ─────┐  │
- Drain          │ │ - Word2Vec          │  │
- IPLoM          │ │ - FastText          │  │
- AEL            │ │ - TF-IDF            │  │
- ┘ └──────────────────────┘  │

- ┘

##### Key Relationships to Highlight
- Bidirectional communication between GUI and Applications Layer
- Data flow from raw logs through processing pipeline
- Demonstrate integration points with external systems
- How different algorithms can be plugged into various layers

**→ Data Flow**
**⟲ Feedback Loop**
**□ Core Component**
**◇ Algorithm Component**
**○ External Interface**
- Optional Connection
- Mandatory Connection

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph GUI["GUI Layer"]
        Dashboard["Dashboard Components"]
        Callbacks["Callbacks"]
        Assets["Assets"]
    end

    subgraph Apps["Applications Layer"]
        AD["Anomaly Detection"]
        CL["Clustering"]
        LS["Log Summarization"]
    end

    subgraph Analysis["Analysis Layer"]
        AnomD["Anomaly Detector"]
        ClustE["Clustering Engine"]
        NND["Neural Network Detector"]
    end

    subgraph InfoEx["Information Extraction Layer"]
        LP["Log Parser"]
        FE["Feature Extractor"]
        LV["Log Vectorizer"]
        CE["Categorical Encoder"]
    end

    subgraph Preproc["Preprocessing Layer"]
        DP["Data Partitioning"]
        DC["Data Cleaning"]
        FS["Format Standardization"]
    end

    subgraph DataLoad["Data Loading Layer"]
        OTM["OpenTelemetry Data Model"]
        CDL["Custom Data Loaders"]
        DMC["Data Model Conversion"]
    end

    subgraph ExtData["External Data Sources"]
        LF["Log Files"]
        SD["Streaming Data"]
        OTS["OpenTelemetry Systems"]
    end

    subgraph AlgoEco["Algorithm Ecosystem"]
        subgraph AnomalyAlgos["Anomaly Detection"]
            SVM["One-Class SVM"]
            IF["Isolation Forest"]
            LSTM["LSTM"]
        end

        subgraph ClusterAlgos["Clustering"]
            KM["K-means"]
            DB["DBSCAN"]
            BR["BIRCH"]
        end

        subgraph ParseAlgos["Parsing"]
            DR["Drain"]
            IP["IPLoM"]
            AE["AEL"]
        end

        subgraph VectAlgos["Vectorization"]
            W2V["Word2Vec"]
            FT["FastText"]
            TF["TF-IDF"]
        end
    end

    ExtData --> DataLoad
    DataLoad --> Preproc
    Preproc --> InfoEx
    InfoEx --> Analysis
    Analysis --> Apps
    Apps --> GUI

    AlgoEco -.-> Analysis
    AlgoEco -.-> InfoEx

    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px
    classDef external fill:#e1d5e7,stroke:#9673a6
    classDef algo fill:#d5e8d4,stroke:#82b366
    classDef core fill:#dae8fc,stroke:#6c8ebf

    class ExtData external
    class AlgoEco algo
    class GUI,Apps,Analysis,InfoEx,Preproc,DataLoad core

    click Dashboard "https://github.com/salesforce/logai/tree/main/gui/pages"
    click Callbacks "https://github.com/salesforce/logai/tree/main/gui/callbacks"
    click Assets "https://github.com/salesforce/logai/tree/main/gui/assets"
    click AD "https://github.com/salesforce/logai/blob/main/logai/applications/log_anomaly_detection.py"
    click CL "https://github.com/salesforce/logai/blob/main/logai/applications/log_clustering.py"
    click LS "https://github.com/salesforce/logai/blob/main/logai/applications/auto_log_summarization.py"
    click AnomD "https://github.com/salesforce/logai/blob/main/logai/analysis/anomaly_detector.py"
    click ClustE "https://github.com/salesforce/logai/blob/main/logai/analysis/clustering.py"
    click NND "https://github.com/salesforce/logai/blob/main/logai/analysis/nn_anomaly_detector.py"
    click LP "https://github.com/salesforce/logai/blob/main/logai/information_extraction/log_parser.py"
    click FE "https://github.com/salesforce/logai/blob/main/logai/information_extraction/feature_extractor.py"
    click LV "https://github.com/salesforce/logai/blob/main/logai/information_extraction/log_vectorizer.py"
    click CE "https://github.com/salesforce/logai/blob/main/logai/information_extraction/categorical_encoder.py"
    click DP "https://github.com/salesforce/logai/blob/main/logai/preprocess/partitioner.py"
    click DC,FS "https://github.com/salesforce/logai/blob/main/logai/preprocess/preprocessor.py"
    click CDL,OTM "https://github.com/salesforce/logai/blob/main/logai/dataloader/data_loader.py"
    click DMC "https://github.com/salesforce/logai/blob/main/logai/dataloader/data_model.py"
    click AnomalyAlgos "https://github.com/salesforce/logai/tree/main/logai/algorithms/anomaly_detection_algo"
    click ClusterAlgos "https://github.com/salesforce/logai/tree/main/logai/algorithms/clustering_algo"
    click ParseAlgos "https://github.com/salesforce/logai/tree/main/logai/algorithms/parsing_algo"
    click VectAlgos "https://github.com/salesforce/logai/tree/main/logai/algorithms/vectorization_algo"
    click LSTM "https://github.com/salesforce/logai/tree/main/logai/algorithms/nn_model"
```
