# helenguohx/logbert architecture

> Architecture diagram of the GitHub repository helenguohx/logbert, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/helenguohx/logbert
- Repository: https://github.com/helenguohx/logbert

## Overview

##### Project Type and Purpose
- This is a research‐oriented ML experimentation framework for log anomaly detection (“LogBERT”) that bundles:
- Data acquisition & preprocessing for three datasets (HDFS, BGL, Thunderbird)
- A core BERT‐based log model (in bert_pytorch/)
- Baseline implementations (DeepLog, LogAnomaly, LogBERT wrappers in each dataset folder)
- Additional tools (logdeep, loglizer, logparser) for parsing, alternative models, and evaluation
- It is **not** a web service; rather it’s a full ML pipeline: raw logs → structured sequences → model training → inference → evaluation.

##### Main Architectural Components
A. External Data Sources & Scripts
- Scripts in scripts/ download raw logs from online repositories
B. Persistent Storage
- ~/.dataset (raw logs)
- project/output (intermediate data, trained models, results)
C. Parsing & Preprocessing
- logparser/ (Drain, Spell parsers)
- data_process.py in each dataset folder
D. Dataset Management
- bert_pytorch/dataset/*
- logdeep/dataset/*
- loglizer/preprocessing.py
E. Model Implementations
- bert_pytorch/model (BERT encoder, attention, embedding, transformer layers)
- logdeep/models (LSTM)
- loglizer/models (DecisionTree, DeepLog, PCA, etc.)
F. Training & Inference Engines
- bert_pytorch/trainer (optim_schedule, pretrain)
- bert_pytorch/train_log.py / predict_log.py
- dataset‐specific wrappers (e.g. HDFS/logbert.py, deeplog.py, loganomaly.py)
- logdeep/tools/train.py / predict.py
G. Evaluation & Visualization
- baselines.ipynb in each dataset folder
- project/output logs and metrics

##### Relationships & Dataflow
1. **Data Download** → scripts/download_*.sh
2. Raw logs stored under `~/.dataset/`
3. **Log Parsing** → logparser/ → structured events
4. **Dataset‐Specific Preprocessing**
- HDFS/data_process.py et al. transform parsed events into sequences
- Results dumped to project/output/<dataset>/
5. **Data Loading**
- bert_pytorch/dataset & logdeep/dataset load sequences → PyTorch DataLoader
6. **Model Training**
- Trainer modules consume DataLoader + model definitions → checkpoints in project/output/
7. **Inference / Prediction**
- predict_log.py or dataset wrappers (logbert.py predict) → anomaly scores
8. **Evaluation**
- baselines.ipynb aggregates metrics and plots

##### Architectural Patterns & Design Principles
- **Modular Separation** between:
- Data acquisition/parsing layer
- Dataset‐agnostic loaders & utilities (in common packages)
- Model implementations (encapsulation of BERT, LSTM, classical ML)
- Training/inference wrappers per dataset (thin CLI interfaces)
- **Reusability & Extensibility**: add a new dataset by copying a folder template; add a new model by dropping in a new module under `bert_pytorch/model` or `loglizer/models`.
- **Pipeline Pattern**: linear flow from raw data → processed sequences → model → evaluation.

##### Essential Details to Capture
- Explicitly show how each dataset folder (HDFS/BGL/TBird) uses shared core libraries (bert_pytorch, logdeep, loglizer)
- Where intermediate CSV/JSON sequence files live
- Trainer/optim schedule in `bert_pytorch/trainer` as a reusable component
- Baseline vs. LogBERT model paths side‐by‐side for comparison

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Acquisition
    subgraph "External Data Acquisition"
        External["GitHub Loghub"]:::cloud
        Download["download_*.sh"]:::script
    end
    External --> Download
    Download --> Raw["~/.dataset"]:::data

    %% Parsing & Preprocessing
    subgraph "Parsing & Preprocessing"
        Parser["logparser/"]:::code
    end
    Raw --> Parser
    Parser --> Parsed["Structured Logs"]:::data

    %% Dataset-specific Preprocessing
    subgraph "Dataset-specific Preprocessing"
        HDFS_PP["HDFS/data_process.py"]:::script
        BGL_PP["BGL/data_process.py"]:::script
        TB_PP["TBird/data_process.py"]:::script
    end
    Parsed --> HDFS_PP --> SeqH["project/output/HDFS/sequences"]:::data
    Parsed --> BGL_PP --> SeqB["project/output/BGL/sequences"]:::data
    Parsed --> TB_PP --> SeqT["project/output/TBird/sequences"]:::data

    %% Dataset Management & Loading
    subgraph "Dataset Management & Loading"
        BertDataset["bert_pytorch/dataset/"]:::code
        LDataset["logdeep/dataset/"]:::code
        LZPre["loglizer/preprocessing.py"]:::script
    end
    SeqH --> BertDataset
    SeqB --> BertDataset
    SeqT --> BertDataset

    SeqH --> LDataset
    SeqB --> LDataset
    SeqT --> LDataset

    SeqH --> LZPre
    SeqB --> LZPre
    SeqT --> LZPre

    %% Model Implementations
    subgraph "Model Implementations"
        BERTModel["bert_pytorch/model/"]:::code
        LSTMModel["logdeep/models/lstm.py"]:::code
        ClassModels["loglizer/models/"]:::code
    end
    BertDataset --> BERTModel
    LDataset --> LSTMModel
    LZPre --> ClassModels

    %% Training & Inference Engines
    subgraph "Training & Inference Engines"
        BERTTrainEngine["bert_pytorch/trainer/"]:::code
        TrainLog["bert_pytorch/train_log.py"]:::script
        PredictLog["bert_pytorch/predict_log.py"]:::script
        HDFS_LB["HDFS/logbert.py"]:::script
        HDFS_DL["HDFS/deeplog.py"]:::script
        HDFS_LA["HDFS/loganomaly.py"]:::script
        BGL_LB["BGL/logbert.py"]:::script
        BGL_DL["BGL/deeplog.py"]:::script
        BGL_LA["BGL/loganomaly.py"]:::script
        TB_LB["TBird/logbert.py"]:::script
        TB_DL["TBird/deeplog.py"]:::script
        TB_LA["TBird/loganomaly.py"]:::script
        LDTrain["logdeep/tools/train.py"]:::script
        LDPred["logdeep/tools/predict.py"]:::script
    end
    BERTModel --> BERTTrainEngine --> TrainLog --> ModelOut["project/output/models"]:::data
    PredictLog --> Scores["Anomaly Scores"]:::data
    LSTMModel --> LDTrain --> LDScore["Anomaly Scores"]:::data
    ClassModels --> LDTrain
    LDTrain --> LDPred --> LDScore
    SeqH --> HDFS_LB --> Scores
    SeqH --> HDFS_DL --> Scores
    SeqH --> HDFS_LA --> Scores
    SeqB --> BGL_LB --> Scores
    SeqB --> BGL_DL --> Scores
    SeqB --> BGL_LA --> Scores
    SeqT --> TB_LB --> Scores
    SeqT --> TB_DL --> Scores
    SeqT --> TB_LA --> Scores

    %% Evaluation & Visualization
    subgraph "Evaluation & Visualization"
        HDFS_E["HDFS/baselines.ipynb"]:::notebook
        BGL_E["BGL/baselines.ipynb"]:::notebook
        TB_E["TBird/baselines.ipynb"]:::notebook
    end
    Scores --> HDFS_E
    Scores --> BGL_E
    Scores --> TB_E

    %% Click Events
    click Download "https://github.com/helenguohx/logbert/blob/main/scripts/download_*.sh"
    click Parser "https://github.com/helenguohx/logbert/tree/main/logparser/"
    click HDFS_PP "https://github.com/helenguohx/logbert/blob/main/HDFS/data_process.py"
    click BGL_PP "https://github.com/helenguohx/logbert/blob/main/BGL/data_process.py"
    click TB_PP "https://github.com/helenguohx/logbert/blob/main/TBird/data_process.py"
    click BertDataset "https://github.com/helenguohx/logbert/tree/main/bert_pytorch/dataset/"
    click LDataset "https://github.com/helenguohx/logbert/tree/main/logdeep/dataset/"
    click LZPre "https://github.com/helenguohx/logbert/blob/main/loglizer/preprocessing.py"
    click BERTModel "https://github.com/helenguohx/logbert/tree/main/bert_pytorch/model/"
    click LSTMModel "https://github.com/helenguohx/logbert/blob/main/logdeep/models/lstm.py"
    click ClassModels "https://github.com/helenguohx/logbert/tree/main/loglizer/models/"
    click BERTTrainEngine "https://github.com/helenguohx/logbert/tree/main/bert_pytorch/trainer/"
    click TrainLog "https://github.com/helenguohx/logbert/blob/main/bert_pytorch/train_log.py"
    click PredictLog "https://github.com/helenguohx/logbert/blob/main/bert_pytorch/predict_log.py"
    click HDFS_LB "https://github.com/helenguohx/logbert/blob/main/HDFS/logbert.py"
    click HDFS_DL "https://github.com/helenguohx/logbert/blob/main/HDFS/deeplog.py"
    click HDFS_LA "https://github.com/helenguohx/logbert/blob/main/HDFS/loganomaly.py"
    click BGL_LB "https://github.com/helenguohx/logbert/blob/main/BGL/logbert.py"
    click BGL_DL "https://github.com/helenguohx/logbert/blob/main/BGL/deeplog.py"
    click BGL_LA "https://github.com/helenguohx/logbert/blob/main/BGL/loganomaly.py"
    click TB_LB "https://github.com/helenguohx/logbert/blob/main/TBird/logbert.py"
    click TB_DL "https://github.com/helenguohx/logbert/blob/main/TBird/deeplog.py"
    click TB_LA "https://github.com/helenguohx/logbert/blob/main/TBird/loganomaly.py"
    click LDTrain "https://github.com/helenguohx/logbert/blob/main/logdeep/tools/train.py"
    click LDPred "https://github.com/helenguohx/logbert/blob/main/logdeep/tools/predict.py"
    click HDFS_E "https://github.com/helenguohx/logbert/blob/main/HDFS/baselines.ipynb"
    click BGL_E "https://github.com/helenguohx/logbert/blob/main/BGL/baselines.ipynb"
    click TB_E "https://github.com/helenguohx/logbert/blob/main/TBird/baselines.ipynb"

    %% Styles
    classDef cloud fill:#b3e5fc,stroke:#0288d1,color:#01579b;
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1,shape:cylinder;
    classDef code fill:#c8e6c9,stroke:#43a047,color:#1b5e20;
    classDef script fill:#fff9c4,stroke:#fbc02d,color:#f57f17;
    classDef parsing fill:#e1bee7,stroke:#8e24aa,color:#4a148c;
    classDef preproc fill:#ffe0b2,stroke:#fb8c00,color:#e65100;
    classDef dataset fill:#d1c4e9,stroke:#5e35b1,color:#311b92;
    classDef model fill:#c5cae9,stroke:#3949ab,color:#1a237e;
    classDef train fill:#dcedc8,stroke:#7cb342,color:#33691e;
    classDef eval fill:#f8bbd0,stroke:#d81b60,color:#880e4f;
    classDef notebook fill:#f3e5f5,stroke:#ab47bc,color:#4a148c;
```
