# logpai/drain3 architecture

> Architecture diagram of the GitHub repository logpai/drain3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/logpai/drain3
- Repository: https://github.com/logpai/drain3

## Overview

##### Project type and purpose
- This is an open-source Python library (CLI tools + API) for online log-template mining.
- It reads raw log messages, extracts/masks variable parts, learns or matches templates on the fly, and persists its internal state.

##### Top-level modules and directory structure
- drain3/           → core package
- drain.py                 → main prefix-tree implementation (Drain algorithm)
- template_miner.py        → high-level API (TemplateMiner)
- persistence_handler.py   → abstract persistence interface
- *‌_persistence.py        → concrete handlers: File, Kafka, Redis, MemoryBuffer
- masking.py               → regex-based masking preprocessor
- simple_profiler.py       → optional time profiler
- template_miner_config.py → config object model
- examples/         → CLI demos (stdin and bigfile)
- tests/            → unit tests
- pyproject.toml    → build and dependencies
- CI & scripts      → GitHub Actions CI, deployment script

##### Core components to model
a. TemplateMiner (API entrypoint)
b. Drain (prefix tree engine + clustering logic)
c. Masking pipeline (RegexMaskingInstruction)
d. PersistenceHandler interface + its concrete implementations (File, Kafka, Redis, Memory)
e. Configuration source (.ini parser or TemplateMinerConfig)
f. ParameterExtraction module
g. External services: Kafka cluster, Redis server, local filesystem
h. Optional Profiler

##### Relationships and data flow
1. Configuration (ini file or object) → TemplateMinerConfig
2. Incoming log message
→ Masking module (tokenize & mask numbers, IPs, etc.)
→ TemplateMiner.add_log_message()
→ Drain.parse_tree (cluster match or creation)
→ (Optional) Profiler times each step
→ PersistenceHandler.snapshot(state) on events
→ Output: JSON-style dictionary with change_type, cluster_id, template, parameters
- In inference mode: TemplateMiner.match() bypasses tree update
3. PersistenceHandler
– Strategy pattern: single abstract interface with .save() / .load()
– FilePersistence writes to disk
– KafkaPersistence publishes to a Kafka topic (requires kafka-python)
– RedisPersistence writes to a Redis key (requires redis-py)
– MemoryBufferPersistence holds snapshot in memory

##### Architectural patterns & design principles
- Strategy pattern for persistence backends
- Adapter pattern to integrate external queues/datastores (Kafka, Redis)
- Separation of concerns:
- Masking (preprocessing)
- Clustering & templating (core algorithm)
- State management & persistence
- Configuration management
- LRU cache eviction for memory-bound clustering
- Extensibility: new persistence or masking can be plugged by inheriting the base class

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI examples
    subgraph "CLI Examples" 
        StdinDemo["stdin Demo"]:::external
        BigfileDemo["bigfile Demo"]:::external
    end

    %% Configuration
    ConfigIni["drain3.ini"]:::config
    ConfigObj["TemplateMinerConfig"]:::config

    %% Core Library
    subgraph "TemplateMiner Library"
        Masking["RegexMaskingInstruction"]:::pre
        Profiler["SimpleProfiler"]:::profiler
        TemplateMiner["TemplateMiner (API)"]:::core
        subgraph "Clustering Engines"
            Drain["Drain"]:::core
            Jaccard["JaccardDrain"]:::core
        end
        subgraph "Persistence Layer"
            PH["PersistenceHandler"]:::persistence
            FileP["FilePersistence"]:::persistence
            KafkaP["KafkaPersistence"]:::persistence
            RedisP["RedisPersistence"]:::persistence
            MemP["MemoryBufferPersistence"]:::persistence
        end
    end

    %% External Services
    KafkaService["Kafka Cluster"]:::external
    RedisService["Redis Server"]:::external
    FileSystem["Local Filesystem"]:::external

    %% Data Flow
    ConfigIni --> ConfigObj
    ConfigObj --> TemplateMiner
    Masking --> TemplateMiner
    TemplateMiner --> Drain
    TemplateMiner --> Jaccard
    TemplateMiner --> Profiler
    TemplateMiner --> PH
    PH --> FileP
    PH --> KafkaP
    PH --> RedisP
    PH --> MemP
    KafkaP --> KafkaService
    RedisP --> RedisService
    FileP --> FileSystem
    StdinDemo --> TemplateMiner
    BigfileDemo --> TemplateMiner

    %% Click Events
    click TemplateMiner "https://github.com/logpai/drain3/blob/master/drain3/template_miner.py"
    click Drain "https://github.com/logpai/drain3/blob/master/drain3/drain.py"
    click Jaccard "https://github.com/logpai/drain3/blob/master/drain3/jaccard_drain.py"
    click Masking "https://github.com/logpai/drain3/blob/master/drain3/masking.py"
    click PH "https://github.com/logpai/drain3/blob/master/drain3/persistence_handler.py"
    click FileP "https://github.com/logpai/drain3/blob/master/drain3/file_persistence.py"
    click KafkaP "https://github.com/logpai/drain3/blob/master/drain3/kafka_persistence.py"
    click RedisP "https://github.com/logpai/drain3/blob/master/drain3/redis_persistence.py"
    click MemP "https://github.com/logpai/drain3/blob/master/drain3/memory_buffer_persistence.py"
    click ConfigObj "https://github.com/logpai/drain3/blob/master/drain3/template_miner_config.py"
    click Profiler "https://github.com/logpai/drain3/blob/master/drain3/simple_profiler.py"
    click StdinDemo "https://github.com/logpai/drain3/blob/master/examples/drain_stdin_demo.py"
    click BigfileDemo "https://github.com/logpai/drain3/blob/master/examples/drain_bigfile_demo.py"
    click ConfigIni "https://github.com/logpai/drain3/blob/master/examples/drain3.ini"

    %% Styles
    classDef core fill:#B3DDF2,stroke:#333,stroke-width:1px
    classDef pre fill:#C2F0C2,stroke:#333,stroke-width:1px
    classDef persistence fill:#FFF2B2,stroke:#333,stroke-width:1px
    classDef config fill:#F9D5E5,stroke:#333,stroke-width:1px
    classDef profiler fill:#E0E0E0,stroke:#333,stroke-width:1px
    classDef external fill:#F8D7DA,stroke:#333,stroke-width:1px
```
