# wuyifan18/adaparser architecture

> Architecture diagram of the GitHub repository wuyifan18/adaparser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wuyifan18/adaparser
- Repository: https://github.com/wuyifan18/adaparser

## Overview

##### Project Type and Purpose
- This is an open-source research tool for log parsing experiments.
- It provides two main functions:
– Parsing logs with an LLM-based parser (AdaParser)
– Evaluating the parsed results (accuracy metrics)

##### Main Components
A. Data Layer
– full_dataset/ (unzipped Loghub-2.0 raw logs)
– sampled_examples_* subdirectories (pre-sampled log snippets)
B. Parsing Service
– benchmark/logparser/AdaParser/AdaParser.py (core parsing logic)
– Trie.py, post_process.py, utils.py (supporting code)
– openai_key.txt → holds API key for invoking OpenAI LLM
C. Orchestration (CLI Entry Point)
– benchmark/evaluation/AdaParser_eval.py
- Parses CLI args: --model, --log_ratio
- Loads sampled logs, invokes AdaParser modules
- Calls evaluation utilities
D. Evaluation Service
– benchmark/evaluation/utils/
- PA_calculator.py (Parsing Accuracy)
- GA_calculator.py (Grouping Accuracy)
- template_level_analysis.py, postprocess.py
- evaluator_main.py (ties calculators together)
E. Persistence / Output
– result/ directory (not in repo, created at runtime)
– Stores parsed logs, metrics, and summary reports
F. External Dependency
– OpenAI API (LLM endpoint)

##### Relationships & Data Flows
1. CLI invocation
→ reads openai_key.txt
→ reads sampled logs from full_dataset
→ calls AdaParser (Parsing Service)
2. AdaParser
→ constructs prompts (self-generated in-context examples)
→ sends to OpenAI API → receives parsed templates
→ applies post_process.py & Trie matching
3. Parsed output
→ fed into Evaluation Service (GA & PA calculators)
4. Evaluation results & artifacts
→ written to result/

##### Architectural Patterns & Principles
- Layered architecture: Data → Parsing → Evaluation → Persistence
- Single‐process, CLI‐driven orchestration
- Clear separation of concerns:
– “logparser” package for parsing logic
– “evaluation” package for metrics

##### Technologies & Frameworks
- Python 3.8+
- OpenAI Python SDK (via requests or openai library)
- Standard Python packages (json, argparse, etc.)
- Requirements.txt lists specific deps

##### Detail Checklist
[ ] Callouts for key scripts (AdaParser_eval.py, AdaParser.py, GA_calculator.py)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Layer"
        D1[(full_dataset/)]:::data
    end

    config["openai_key.txt"]:::config
    requirements["requirements.txt"]:::config
    license["LICENSE"]:::config
    readme["README.md"]:::config

    subgraph "Parsing Service"
        AP["AdaParser.py"]:::parsing
        Trie["Trie.py"]:::parsing
        PP["post_process.py"]:::parsing
        Utils["utils.py"]:::parsing
        Common["common.json"]:::parsing
    end

    subgraph "Orchestration (CLI)"
        Orch["AdaParser_eval.py"]:::orchestration
    end

    subgraph "Evaluation Service"
        GA["GA_calculator.py"]:::evaluation
        PA["PA_calculator.py"]:::evaluation
        TLA["template_level_analysis.py"]:::evaluation
        EP["postprocess.py"]:::evaluation
        EM["evaluator_main.py"]:::evaluation
    end

    subgraph "Persistence"
        D2[(result/)]:::data
    end

    API["OpenAI API"]:::external

    D1 -->|reads logs| Orch
    Orch -->|reads key| config
    Orch -->|parses via| AP
    AP -->|uses| Trie
    AP -->|uses| PP
    AP -->|uses| Utils
    AP -->|reads config| Common
    AP -->|sends prompt| API
    API -->|returns parse| AP
    AP -->|outputs parsed| EM
    EM -->|invokes| GA
    EM -->|invokes| PA
    EM -->|analysis| TLA
    EM -->|postprocess| EP
    EM -->|writes metrics| D2

    classDef data fill:#D6EAF8,stroke:#3498DB,color:#154360;
    classDef parsing fill:#D5F5E3,stroke:#27AE60,color:#145A32;
    classDef evaluation fill:#FDEBD0,stroke:#E67E22,color:#873600;
    classDef orchestration fill:#FADBD8,stroke:#EC7063,color:#641E16;
    classDef external fill:#F0F3F4,stroke:#7F8C8D,color:#17202A;
    classDef config fill:#EBDEF0,stroke:#9B59B6,color:#4A235A;

    click D1 "https://github.com/wuyifan18/adaparser/tree/main/full_dataset/"
    click AP "https://github.com/wuyifan18/adaparser/blob/main/benchmark/logparser/AdaParser/AdaParser.py"
    click Trie "https://github.com/wuyifan18/adaparser/blob/main/benchmark/logparser/AdaParser/Trie.py"
    click PP "https://github.com/wuyifan18/adaparser/blob/main/benchmark/logparser/AdaParser/post_process.py"
    click Utils "https://github.com/wuyifan18/adaparser/blob/main/benchmark/logparser/AdaParser/utils.py"
    click Common "https://github.com/wuyifan18/adaparser/blob/main/benchmark/logparser/AdaParser/common.json"
    click config "https://github.com/wuyifan18/adaparser/blob/main/openai_key.txt"
    click Orch "https://github.com/wuyifan18/adaparser/blob/main/benchmark/evaluation/AdaParser_eval.py"
    click GA "https://github.com/wuyifan18/adaparser/blob/main/benchmark/evaluation/utils/GA_calculator.py"
    click PA "https://github.com/wuyifan18/adaparser/blob/main/benchmark/evaluation/utils/PA_calculator.py"
    click TLA "https://github.com/wuyifan18/adaparser/blob/main/benchmark/evaluation/utils/template_level_analysis.py"
    click EP "https://github.com/wuyifan18/adaparser/blob/main/benchmark/evaluation/utils/postprocess.py"
    click EM "https://github.com/wuyifan18/adaparser/blob/main/benchmark/evaluation/utils/evaluator_main.py"
    click requirements "https://github.com/wuyifan18/adaparser/blob/main/requirements.txt"
    click license "https://github.com/wuyifan18/adaparser/tree/main/LICENSE"
    click readme "https://github.com/wuyifan18/adaparser/blob/main/README.md"
```
