How helenguohx/logbert works
Project Type and Purpose
• This is a research‐oriented ML experimentation framework for log anomaly detection (“LogBERT”) that bundles:
• Data acquisition & preprocessing for three datasets (HDFS, BGL, Thunderbird)
• A core BERT‐based log model (in bert_pytorch/)
• Baseline implementations (DeepLog, LogAnomaly, LogBERT wrappers in each dataset folder)
• Additional tools (logdeep, loglizer, logparser) for parsing, alternative models, and evaluation
• It is not a web service; rather it’s a full ML pipeline: raw logs → structured sequences → model training → inference → evaluation.
Main Architectural Components
A. External Data Sources & Scripts
• Scripts in scripts/ download raw logs from online repositories
B. Persistent Storage
• ~/.dataset (raw logs)
• project/output (intermediate data, trained models, results)
C. Parsing & Preprocessing
• logparser/ (Drain, Spell parsers)
• data_process.py in each dataset folder
D. Dataset Management
• bert_pytorch/dataset/*
• logdeep/dataset/*
• loglizer/preprocessing.py
E. Model Implementations
• bert_pytorch/model (BERT encoder, attention, embedding, transformer layers)
• logdeep/models (LSTM)
• loglizer/models (DecisionTree, DeepLog, PCA, etc.)
F. Training & Inference Engines
• bert_pytorch/trainer (optim_schedule, pretrain)
• bert_pytorch/train_log.py / predict_log.py
• dataset‐specific wrappers (e.g. HDFS/logbert.py, deeplog.py, loganomaly.py)
• logdeep/tools/train.py / predict.py
G. Evaluation & Visualization
• baselines.ipynb in each dataset folder
• project/output logs and metrics
Relationships & Dataflow
1. Data Download → scripts/download_*.sh
2. Raw logs stored under ~/.dataset/
3. Log Parsing → logparser/ → structured events
4. Dataset‐Specific Preprocessing
• HDFS/data_process.py et al. transform parsed events into sequences
• Results dumped to project/output/<dataset>/
5. Data Loading
• bert_pytorch/dataset & logdeep/dataset load sequences → PyTorch DataLoader
6. Model Training
• Trainer modules consume DataLoader + model definitions → checkpoints in project/output/
7. Inference / Prediction
• predict_log.py or dataset wrappers (logbert.py predict) → anomaly scores
8. Evaluation
• baselines.ipynb aggregates metrics and plots
Architectural Patterns & Design Principles
• Modular Separation between:
• Data acquisition/parsing layer
• Dataset‐agnostic loaders & utilities (in common packages)
• Model implementations (encapsulation of BERT, LSTM, classical ML)
• Training/inference wrappers per dataset (thin CLI interfaces)
• Reusability & Extensibility: add a new dataset by copying a folder template; add a new model by dropping in a new module under bert_pytorch/model or loglizer/models.
• Pipeline Pattern: linear flow from raw data → processed sequences → model → evaluation.
Essential Details to Capture
• Explicitly show how each dataset folder (HDFS/BGL/TBird) uses shared core libraries (bert_pytorch, logdeep, loglizer)
• Where intermediate CSV/JSON sequence files live
• Trainer/optim schedule in bert_pytorch/trainer as a reusable component
• Baseline vs. LogBERT model paths side‐by‐side for comparison