# giuliomat95/pydeeplog architecture

> Architecture diagram of the GitHub repository giuliomat95/pydeeplog, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/giuliomat95/pydeeplog
- Repository: https://github.com/giuliomat95/pydeeplog

## Overview

##### Project type and purpose
- DeepLog Trainer is a Python-based data-processing and machine-learning pipeline (“open-source tool”) designed to parse raw system logs and train three types of models (log-key anomaly detection, parameter-value anomaly detection, and workflow modeling) to detect and diagnose anomalies in those logs.

##### File-structure overview
- Top-level directories/files:
- data/ – sample logs & datasets
- deeplog_trainer/ – core library
– log_parser/ (Drain3 adapter, sessions)
– model/ (data preprocessing, training, evaluation, model management)
– parameter_detection/ (parallel structure for parameter anomaly)
– workflow/ (workflow FSM construction and network utilities)
- run/ – CLI entrypoints for each pipeline stage (run_deeplog, run_drain, run_log_key_detection_model, run_parameter_detection_model, run_workflow)
- config (hdfs_config.ini & others) for Drain parameters
- Dockerfile / requirements.txt / pytest tests

##### Main components & relationships
A. Ingestion & Configuration
- Raw Log Files (data/*.log, data/dataset.json)
- Config Loader (hdfs_config.ini or other .ini)
→ feeds parameters (similarity thresholds, session delimiters, LSTM hyperparameters) into downstream stages

B. Log-Parsing Stage
- Drain3 Parser (deeplog_trainer/log_parser/drain.py)
- Adapter (sessionizing logs, deeplog_trainer/log_parser/adapter.py)
- Session Builder (deeplog_trainer/log_parser/sessions.py)
→ outputs structured sequences of (log_key_id, parameter_vector, timestamps)

C. Log-Key Anomaly Detection Model
- Data Preprocessing (deeplog_trainer/model/data_preprocess.py)
- LSTM Training (deeplog_trainer/model/training.py, parameters: window_size, lstm_units, etc.)
- Model Manager / Evaluator (model_manager.py, model_evaluator.py)
→ produces logkey_model.h5, metrics, tensorboard logs

D. Parameter-Value Anomaly Detection Model
- Similar pipeline under deeplog_trainer/parameter_detection:
data_preprocess.py, model_evaluator.py
- LSTM forecasting on numerical parameter vectors + time deltas
→ produces log_par_model.h5, confidence intervals, metrics

E. Workflow Modeling
- Sequence clustering and FSM construction (deeplog_trainer/workflow/network.py, workflow.py)
- Detects concurrency divergences and loops
→ outputs workflows as pickle files

F. Orchestration & CLI
- run/ scripts wire together the above stages in configurable pipelines
- run_deeplog.py ties parsing, log-key model, parameter model, and workflow model into one command

G. Artifacts & External Services
- Artifacts Storage (artifacts/… for models, tensorboard dirs)
- Tensorboard for training visualization
- Docker container for reproducible environment
- Pytest for continuous integration

##### Architectural patterns & design principles
- Pipeline / staged ETL + ML training
- Modular, layered design (parser → model → evaluation → management)
- Pluggable components (e.g., different adapter types, thresholds)
- Separation of concerns: parsing, model training, workflow analysis

##### Detail level
- Interactions (e.g., how model_evaluator reads model outputs to compute metrics)
- Don’t omit the test suite – you may group all tests under a “CI / Testing” container

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Stores
    RawLogs((Raw Log Files)):::datastore
    ConfigLoader((Config Loader)):::datastore

    %% Log Parsing Stage
    subgraph "Log Parsing Stage"
        direction TB
        DrainParser["Drain3 Parser"]:::parser
        Adapter["Log-Parser Adapter"]:::parser
        Sessions["Session Builder"]:::parser
    end

    SessionData((Parsed Sessions)):::datastore

    %% Log-Key Anomaly Detection Pipeline
    subgraph "Log-Key Anomaly Detection" 
        direction TB
        LogKeyDataPrep["Data Preprocessing"]:::ml
        LogKeyLSTM["LSTM Training"]:::ml
        LogKeyManager["Model Manager"]:::ml
        LogKeyEvaluator["Model Evaluator"]:::ml
    end
    LogKeyArtifact((logkey_model.h5 + metrics)):::datastore

    %% Parameter-Value Anomaly Detection Pipeline
    subgraph "Parameter-Value Anomaly Detection"
        direction TB
        ParamDataPrep["Data Preprocessing"]:::ml
        ParamEvaluator["Model Evaluator"]:::ml
    end
    ParamArtifact((log_par_model.h5 + metrics)):::datastore

    %% Workflow Modeling Pipeline
    subgraph "Workflow Modeling"
        direction TB
        WorkflowNetwork["Network / FSM Construction"]:::workflow
        WorkflowUtils["Workflow Utilities"]:::workflow
    end
    WorkflowArtifact((Workflow Pickles)):::datastore

    %% Orchestration & CLI
    subgraph "Orchestration & CLI"
        direction TB
        CLIFull["Full Pipeline CLI"]:::cli
        CLIDrain["Drain Stage CLI"]:::cli
        CLILogKey["Log-Key Model CLI"]:::cli
        CLIParam["Parameter Model CLI"]:::cli
        CLIWorkflow["Workflow Stage CLI"]:::cli
    end

    %% External & CI
    Docker[("Docker Container")]:::external
    Requirements[("Python Dependencies")]:::external
    Tensorboard(("Tensorboard")):::external
    Tests(("CI / Testing")):::external

    %% Connections
    RawLogs -->|reads| DrainParser
    ConfigLoader -->|hyperparameters| DrainParser
    DrainParser --> Adapter
    Adapter --> Sessions
    Sessions --> SessionData

    SessionData -->|training sequences| LogKeyDataPrep
    LogKeyDataPrep --> LogKeyLSTM
    LogKeyLSTM --> LogKeyManager
    LogKeyManager --> LogKeyEvaluator
    LogKeyEvaluator --> LogKeyArtifact

    SessionData -->|parameter vectors| ParamDataPrep
    ParamDataPrep --> ParamEvaluator
    ParamEvaluator --> ParamArtifact

    SessionData -->|event sequences| WorkflowNetwork
    WorkflowNetwork --> WorkflowUtils
    WorkflowUtils --> WorkflowArtifact

    CLIFull --> DrainParser
    CLIFull --> LogKeyLSTM
    CLIFull --> ParamEvaluator
    CLIFull --> WorkflowUtils

    CLIDrain --> DrainParser
    CLILogKey --> LogKeyLSTM
    CLIParam --> ParamEvaluator
    CLIWorkflow --> WorkflowUtils

    Docker --- CLIFull
    Docker --- Tensorboard
    Requirements --- CLIFull
    Tests --- DrainParser
    Tests --- LogKeyLSTM
    Tests --- ParamEvaluator
    Tests --- WorkflowUtils

    %% Click Events
    click RawLogs "https://github.com/giuliomat95/pydeeplog/tree/master/data/"
    click ConfigLoader "https://github.com/giuliomat95/pydeeplog/blob/master/hdfs_config.ini"
    click DrainParser "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/log_parser/drain.py"
    click Adapter "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/log_parser/adapter.py"
    click Sessions "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/log_parser/sessions.py"
    click LogKeyDataPrep "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/model/data_preprocess.py"
    click LogKeyLSTM "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/model/training.py"
    click LogKeyManager "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/model/model_manager.py"
    click LogKeyEvaluator "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/model/model_evaluator.py"
    click ParamDataPrep "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/parameter_detection/data_preprocess.py"
    click ParamEvaluator "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/parameter_detection/model_evaluator.py"
    click WorkflowNetwork "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/workflow/network.py"
    click WorkflowUtils "https://github.com/giuliomat95/pydeeplog/blob/master/deeplog_trainer/workflow/workflow.py"
    click CLIFull "https://github.com/giuliomat95/pydeeplog/blob/master/run/run_deeplog.py"
    click CLIDrain "https://github.com/giuliomat95/pydeeplog/blob/master/run/run_drain.py"
    click CLILogKey "https://github.com/giuliomat95/pydeeplog/blob/master/run/run_log_key_detection_model.py"
    click CLIParam "https://github.com/giuliomat95/pydeeplog/blob/master/run/run_parameter_detection_model.py"
    click CLIWorkflow "https://github.com/giuliomat95/pydeeplog/blob/master/run/run_workflow.py"
    click Docker "https://github.com/giuliomat95/pydeeplog/tree/master/Dockerfile"
    click Requirements "https://github.com/giuliomat95/pydeeplog/blob/master/requirements.txt"
    click Tests "https://github.com/giuliomat95/pydeeplog/tree/master/tests/"

    %% Styles
    classDef parser fill:#ADD8E6,stroke:#000,color:#000
    classDef ml fill:#90EE90,stroke:#000,color:#000
    classDef workflow fill:#FFA500,stroke:#000,color:#000
    classDef datastore fill:#D3D3D3,stroke:#000,color:#000,stroke-dasharray: 5 5
    classDef cli fill:#DDA0DD,stroke:#000,color:#000
    classDef external fill:#F0E68C,stroke:#000,color:#000,stroke-dasharray: 2 2
```
