# ye-kyaw-thu/khpos architecture

> Architecture diagram of the GitHub repository ye-kyaw-thu/khpos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ye-kyaw-thu/khpos
- Repository: https://github.com/ye-kyaw-thu/khpos

## Overview

##### Project type and purpose
- This is not a web or mobile app but a research‐oriented data‐processing and machine-learning pipeline.
- Purpose: take a manually segmented and tagged Khmer corpus, run preprocessing, train six incremental POS-tagging models (2-hours, 3gHMM, CRF, Kytea SVM, MaxEnt, RDR), evaluate each, and produce accuracy logs.

##### File structure
- Top‐level:
- README.md – project overview
- corpus-draft-ver-1.0/
– data/ – raw data + preprocessing scripts (Perl, shell)
– model/ – six subdirectories, one per tagging method
· 2hours/, 3gHMM/, crf/, kytea/, maxent/, rdr/
· each contains: train/test partitions, training scripts (train-all.sh, etc.), model artifacts, evaluation scripts (eval-*.sh), logs
- No frontend/backend; this is purely a data pipeline with model branches.

##### Key insights from README
- Scripts: mk-wordtag.pl, mk-pair.pl for data formatting
- Six POS tagging approaches, each with incremental training on 2K → 12K sentences
- External dependencies invoked by scripts (CRF++ toolkit, Kytea binary, Perl, possibly Python for cf.py)

##### Main components & interactions
- Raw corpus under data/ (files: OPEN-TEST.word, CLOSE-TEST.tag, trainX.word/tag)
b. Preprocessing Engine (oval):
- Perl scripts (mk-wordtag.pl, mk-pair.pl), shell (split-all.sh, freq.sh)
- Outputs unified word/tag formats and splits (train1…train6 sets)
1. 2-hours pipeline
2. 3gHMM pipeline
3. crf pipeline (invokes CRF++ binaries)
4. kytea pipeline (invokes Kytea binary)
5. maxent pipeline (invokes MaxEnt toolkit)
6. rdr pipeline (invokes RDRTool)
- Each pipeline:
– Training script (train-all.sh)
– Per-iteration data feeds (train1…train6)
– Model artifact outputs (.model, .pipe.ser, etc.)
– Evaluation scripts (eval-*.sh) & grading (gradepos.pl)
d. Evaluation & Metrics Aggregator (diamond):
- Collects logs from all pipelines (final-*-accuracy.log, f1.line, etc.)
- Generates summary comparisons
e. Output Artifacts (rectangle):
- Final models, per-model accuracy reports, charts (optional)

##### Architectural patterns & design principles
- Pipeline pattern: a sequence of processing stages
- Branching/parallelization: identical flow split into six model‐specific pipelines
- Separation of concerns: preprocessing separated from training, training separated from evaluation
- Incremental training: each model pipeline loops over data subsets (2K → 12K)

##### Detail level
- Of the six pipelines separately, listing key scripts in each
- Data splits (train1…train6, OPEN-TEST, CLOSE-TEST) feeding into each pipeline
- Evaluation scripts and log outputs per pipeline
- Do not omit the preprocessing step, as it is critical to data quality

## Diagram (Mermaid)

```mermaid
flowchart TD
    InputData[(Input Data Store)]:::data
    click InputData "https://github.com/ye-kyaw-thu/khpos/tree/master/corpus-draft-ver-1.0/data/"

    Preproc((Preprocessing Engine)):::processing
    InputData --> Preproc

    subgraph "Preprocessing Scripts"
        MT["mk-wordtag.pl"]:::processing
        MP["mk-pair.pl"]:::processing
        SA["split-all.sh"]:::processing
        FR["freq.sh"]:::processing
        TE["tag-error-fix.sh"]:::processing
    end
    Preproc --> MT
    Preproc --> MP
    Preproc --> SA
    Preproc --> FR
    Preproc --> TE

    click MT "https://github.com/ye-kyaw-thu/khpos/blob/master/corpus-draft-ver-1.0/data/mk-wordtag.pl"
    click MP "https://github.com/ye-kyaw-thu/khpos/blob/master/corpus-draft-ver-1.0/data/mk-pair.pl"
    click SA "https://github.com/ye-kyaw-thu/khpos/blob/master/corpus-draft-ver-1.0/data/split-all.sh"
    click FR "https://github.com/ye-kyaw-thu/khpos/blob/master/corpus-draft-ver-1.0/data/freq.sh"
    click TE "https://github.com/ye-kyaw-thu/khpos/blob/master/corpus-draft-ver-1.0/data/tag-error-fix.sh"

    subgraph "Model Pipelines"
        subgraph "2-Hours Pipeline"
            P1Root["2-Hours Pipeline"]:::pipeline1
            click P1Root "https://github.com/ye-kyaw-thu/khpos/tree/master/corpus-draft-ver-1.0/model/2hours/"
        end
        subgraph "3gHMM Pipeline"
            P2Root["3gHMM Pipeline"]:::pipeline2
            click P2Root "https://github.com/ye-kyaw-thu/khpos/tree/master/corpus-draft-ver-1.0/model/3gHMM/"
        end
        subgraph "CRF++ Pipeline"
            P3Root["CRF Pipeline"]:::pipeline3
            click P3Root "https://github.com/ye-kyaw-thu/khpos/tree/master/corpus-draft-ver-1.0/model/crf/"
        end
        subgraph "Kytea Pipeline"
            P4Root["Kytea SVM Pipeline"]:::pipeline4
            click P4Root "https://github.com/ye-kyaw-thu/khpos/tree/master/corpus-draft-ver-1.0/model/kytea/"
        end
        subgraph "MaxEnt Pipeline"
            P5Root["MaxEnt Pipeline"]:::pipeline5
            click P5Root "https://github.com/ye-kyaw-thu/khpos/tree/master/corpus-draft-ver-1.0/model/maxent/"
        end
        subgraph "RDR Pipeline"
            P6Root["RDRTool Pipeline"]:::pipeline6
            click P6Root "https://github.com/ye-kyaw-thu/khpos/tree/master/corpus-draft-ver-1.0/model/rdr/"
        end
    end

    Preproc --> P1Root
    Preproc --> P2Root
    Preproc --> P3Root
    Preproc --> P4Root
    Preproc --> P5Root
    Preproc --> P6Root

    Agg{Evaluation & Metrics Aggregator}:::diamond
    P1Root --> Agg
    P2Root --> Agg
    P3Root --> Agg
    P4Root --> Agg
    P5Root --> Agg
    P6Root --> Agg

    Output[Output Artifacts]:::output
    Agg --> Output

    classDef data fill:#cde,stroke:#036,stroke-width:1px
    classDef processing fill:#ecf,stroke:#036,stroke-width:1px
    classDef pipeline1 fill:#fdd,stroke:#d33,stroke-width:1px
    classDef pipeline2 fill:#dfd,stroke:#3d3,stroke-width:1px
    classDef pipeline3 fill:#ddf,stroke:#33d,stroke-width:1px
    classDef pipeline4 fill:#fec,stroke:#fb3,stroke-width:1px
    classDef pipeline5 fill:#edf,stroke:#63d,stroke-width:1px
    classDef pipeline6 fill:#cfe,stroke:#3b8,stroke-width:1px
    classDef diamond fill:#ffd,stroke:#aa0,stroke-width:1px
    classDef output fill:#cff,stroke:#07a,stroke-width:1px
```
