How ye-kyaw-thu/khpos works
Project type and purpose
• This is not a web or mobile app but a research‐oriented data‐processing and machine-learning pipeline.
• Purpose: take a manually segmented and tagged Khmer corpus, run preprocessing, train six incremental POS-tagging models (2-hours, 3gHMM, CRF, Kytea SVM, MaxEnt, RDR), evaluate each, and produce accuracy logs.
File structure
• Top‐level:
• README.md – project overview
• corpus-draft-ver-1.0/
– data/ – raw data + preprocessing scripts (Perl, shell)
– model/ – six subdirectories, one per tagging method
· 2hours/, 3gHMM/, crf/, kytea/, maxent/, rdr/
· each contains: train/test partitions, training scripts (train-all.sh, etc.), model artifacts, evaluation scripts (eval-*.sh), logs
• No frontend/backend; this is purely a data pipeline with model branches.
Key insights from README
• Scripts: mk-wordtag.pl, mk-pair.pl for data formatting
• Six POS tagging approaches, each with incremental training on 2K → 12K sentences
• External dependencies invoked by scripts (CRF++ toolkit, Kytea binary, Perl, possibly Python for cf.py)
Main components & interactions
• Raw corpus under data/ (files: OPEN-TEST.word, CLOSE-TEST.tag, trainX.word/tag)
b. Preprocessing Engine (oval):
• Perl scripts (mk-wordtag.pl, mk-pair.pl), shell (split-all.sh, freq.sh)
• Outputs unified word/tag formats and splits (train1…train6 sets)
1. 2-hours pipeline
2. 3gHMM pipeline
3. crf pipeline (invokes CRF++ binaries)
4. kytea pipeline (invokes Kytea binary)
5. maxent pipeline (invokes MaxEnt toolkit)
6. rdr pipeline (invokes RDRTool)
• Each pipeline:
– Training script (train-all.sh)
– Per-iteration data feeds (train1…train6)
– Model artifact outputs (.model, .pipe.ser, etc.)
– Evaluation scripts (eval-*.sh) & grading (gradepos.pl)
d. Evaluation & Metrics Aggregator (diamond):
• Collects logs from all pipelines (final-*-accuracy.log, f1.line, etc.)
• Generates summary comparisons
e. Output Artifacts (rectangle):
• Final models, per-model accuracy reports, charts (optional)
Architectural patterns & design principles
• Pipeline pattern: a sequence of processing stages
• Branching/parallelization: identical flow split into six model‐specific pipelines
• Separation of concerns: preprocessing separated from training, training separated from evaluation
• Incremental training: each model pipeline loops over data subsets (2K → 12K)
Detail level
• Of the six pipelines separately, listing key scripts in each
• Data splits (train1…train6, OPEN-TEST, CLOSE-TEST) feeding into each pipeline
• Evaluation scripts and log outputs per pipeline
• Do not omit the preprocessing step, as it is critical to data quality