# kagamibaka/dlol architecture

> Architecture diagram of the GitHub repository kagamibaka/dlol, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kagamibaka/dlol
- Repository: https://github.com/kagamibaka/dlol

## Overview

##### Project type and purpose
- This is a research‐oriented PyTorch codebase implementing “Dynamic Label-Order Learning” for multi-label classification (event extraction) on the Reuters dataset, based on the AAAI’24 paper. It is neither a traditional full-stack app nor a microservice; it’s an ML pipeline/research tool.

##### File structure
- Top-level directories:
- data/ & data_convert/: raw and processed data, conversion scripts
- seq2seq/: model architecture and utilities
- extraction/: post-prediction parsing and metric calculation
- scripts/: auxiliary bash/python preprocessing
- Key files/scripts:
- run_reuters.sh & run_seq2seq*.bash / run_seq2seq.py – orchestrate training/inference
- evaluation.py & run_eval.bash – compute final metrics
- checkpoint-*/ – serialized model and prediction outputs

##### README for insights
- Confirms the pipeline steps: environment setup → data preprocessing → training (run_reuters.sh) → evaluation (run_eval.bash).
- Core frameworks: Python 3.10, PyTorch, CUDA, BART.

##### Main components & their relationships
A. Data Layer
– Raw JSON files: data/reuters/*.json
– Shuffled splits: data/reuters/shuffled/{train,dev,test}.json
B. Data Conversion
– data_convert/ (convert_text_to_tree.py, task_format/, format/)
– scripts/processing_data.bash
→ Outputs model-ready inputs (tree/text formats)
C. Model Training & Inference
– run_reuters.sh invokes run_seq2seq.py / run_seq2seq_verbose.bash
– my_bart.py wraps the BART model
– seq2seq/: constrained_seq2seq.py, label_smoother_sum.py, utils.py, sentence_splitter.py
→ Produces model checkpoints under checkpoint-* and raw prediction files
D. Prediction Parsing & Extraction
– extraction/predict_parser/ (tree_predict_parser.py)
– extraction/extract_constraint.py, label_tree.py, event_schema.py
– extraction/extraction_metrics.py, calc_header_f_score_UFET.py
→ Converts raw seq2seq outputs into structured event labels
E. Evaluation
– evaluation.py + run_eval.bash
– Consumes parsed predictions and gold data → computes final scores (test_results_seq2seq.txt)

##### Architectural patterns & design principles
- Pipeline modularity: clear separation of preprocessing, model, parsing, evaluation
- Reusable scripts to orchestrate each stage
- Directory grouping by functionality (data, model, extraction)

##### Final reminders
- Be highly detailed: include every major submodule and script
- Data formats at each stage (raw JSON → tree format → tokenized input → raw text preds → structured events)
- Don’t overthink: separate the system into these functional components and illustrate their connectivity clearly

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        RawJSON["Raw JSON Files\n(data/reuters/*.json)"]:::data
        SplitJSON["Shuffled Splits\n(data/reuters/shuffled/*.json)"]:::data
        ProcessData["process_data.py"]:::script
    end

    %% Data Conversion Module
    subgraph "Data Conversion Module"
        ConvertTree["convert_text_to_tree.py"]:::compute
        Text2Tree["text2tree.py"]:::compute
        TargetFormat["target_format.py"]:::compute
        EventFormat["event_extraction.py"]:::compute
        DCUtils["utils.py"]:::compute
        ProcScript["processing_data.bash"]:::script
    end

    %% Model Training & Inference Engine
    subgraph "Model Training & Inference"
        ReutersScript["run_reuters.sh"]:::script
        Seq2SeqPy["run_seq2seq.py"]:::script
        Seq2SeqVBash["run_seq2seq_verbose.bash"]:::script
        BARTWrapper["my_bart.py"]:::compute
        ConstrainedSeq["constrained_seq2seq.py"]:::compute
        LabelSmoother["label_smoother_sum.py"]:::compute
        SentenceSplit["sentence_splitter.py"]:::compute
        SeqUtils["utils.py"]:::compute
        Checkpoints["Checkpoint & raw preds\n(checkpoint-*/test_preds_seq2seq.txt)"]:::data
    end

    %% Prediction Parsing & Extraction Module
    subgraph "Prediction Parsing & Extraction"
        TreeParser["tree_predict_parser.py"]:::parse
        ParserMain["predict_parser.py"]:::parse
        Constraint["extract_constraint.py"]:::parse
        LabelTree["label_tree.py"]:::parse
        Schema["event_schema.py"]:::parse
        ExtractMetrics["extraction_metrics.py"]:::parse
        HeaderF["calc_header_f_score_UFET.py"]:::parse
    end

    %% Evaluation Module
    subgraph "Evaluation Module"
        EvalDriver["evaluation.py"]:::script
        EvalScript["run_eval.bash"]:::script
        EvalResults["Test Results\n(test_results_seq2seq.txt)"]:::data
    end

    %% Connections
    RawJSON -->|feeds| ConvertTree
    SplitJSON -->|feeds| ConvertTree
    ProcessData -->|orchestrates| ConvertTree
    ConvertTree -->|outputs token/tree data| ReutersScript
    ProcScript -->|preprocess| ConvertTree

    ReutersScript -->|calls| Seq2SeqPy
    ReutersScript -->|calls| Seq2SeqVBash
    Seq2SeqPy -->|uses| BARTWrapper
    BARTWrapper -->|uses| ConstrainedSeq
    BARTWrapper -->|uses| LabelSmoother
    BARTWrapper -->|uses| SentenceSplit
    BARTWrapper -->|uses| SeqUtils
    Seq2SeqPy -->|produces| Checkpoints

    Checkpoints -->|inputs| TreeParser
    Seq2SeqPy -->|raw preds| ParserMain
    TreeParser -->|parsed output| Constraint
    Constraint --> LabelTree
    LabelTree --> Schema
    Schema --> ExtractMetrics
    ExtractMetrics --> HeaderF
    HeaderF --> EvalDriver

    EvalDriver -->|runs| EvalScript
    EvalScript -->|reads parsed+gold| EvalResults

    %% Click Events: Data Layer
    click RawJSON "https://github.com/kagamibaka/dlol/tree/main/data/reuters/"
    click SplitJSON "https://github.com/kagamibaka/dlol/tree/main/data/reuters/"
    click ProcessData "https://github.com/kagamibaka/dlol/blob/main/data/reuters/process_data.py"

    %% Click Events: Data Conversion
    click ConvertTree "https://github.com/kagamibaka/dlol/blob/main/data_convert/convert_text_to_tree.py"
    click Text2Tree "https://github.com/kagamibaka/dlol/blob/main/data_convert/format/text2tree.py"
    click TargetFormat "https://github.com/kagamibaka/dlol/blob/main/data_convert/format/target_format.py"
    click EventFormat "https://github.com/kagamibaka/dlol/blob/main/data_convert/task_format/event_extraction.py"
    click DCUtils "https://github.com/kagamibaka/dlol/blob/main/data_convert/utils.py"
    click ProcScript "https://github.com/kagamibaka/dlol/blob/main/scripts/processing_data.bash"

    %% Click Events: Model Training & Inference
    click ReutersScript "https://github.com/kagamibaka/dlol/blob/main/run_reuters.sh"
    click Seq2SeqPy "https://github.com/kagamibaka/dlol/blob/main/run_seq2seq.py"
    click Seq2SeqVBash "https://github.com/kagamibaka/dlol/blob/main/run_seq2seq_verbose.bash"
    click BARTWrapper "https://github.com/kagamibaka/dlol/blob/main/my_bart.py"
    click ConstrainedSeq "https://github.com/kagamibaka/dlol/blob/main/seq2seq/constrained_seq2seq.py"
    click LabelSmoother "https://github.com/kagamibaka/dlol/blob/main/seq2seq/label_smoother_sum.py"
    click SentenceSplit "https://github.com/kagamibaka/dlol/blob/main/seq2seq/sentence_splitter.py"
    click SeqUtils "https://github.com/kagamibaka/dlol/blob/main/seq2seq/utils.py"
    click Checkpoints "https://github.com/kagamibaka/dlol/blob/main/checkpoint-23500_eval/test_preds_seq2seq.txt"

    %% Click Events: Prediction Parsing & Extraction
    click TreeParser "https://github.com/kagamibaka/dlol/blob/main/extraction/predict_parser/tree_predict_parser.py"
    click ParserMain "https://github.com/kagamibaka/dlol/blob/main/extraction/predict_parser/predict_parser.py"
    click Constraint "https://github.com/kagamibaka/dlol/blob/main/extraction/extract_constraint.py"
    click LabelTree "https://github.com/kagamibaka/dlol/blob/main/extraction/label_tree.py"
    click Schema "https://github.com/kagamibaka/dlol/blob/main/extraction/event_schema.py"
    click ExtractMetrics "https://github.com/kagamibaka/dlol/blob/main/extraction/extraction_metrics.py"
    click HeaderF "https://github.com/kagamibaka/dlol/blob/main/extraction/calc_header_f_score_UFET.py"

    %% Click Events: Evaluation Module
    click EvalDriver "https://github.com/kagamibaka/dlol/blob/main/evaluation.py"
    click EvalScript "https://github.com/kagamibaka/dlol/blob/main/run_eval.bash"
    click EvalResults "https://github.com/kagamibaka/dlol/blob/main/checkpoint-23500_eval/test_results_seq2seq.txt"

    %% Styles
    classDef data fill:#b3cde0,stroke:#036,stroke-width:1px;
    classDef compute fill:#ccebc5,stroke:#060,stroke-width:1px;
    classDef parse fill:#fdfd96,stroke:#cc0,stroke-width:1px;
    classDef script fill:#fbb4ae,stroke:#c00,stroke-width:1px;
```
