# tramgianguyen/phofrauddetect architecture

> Architecture diagram of the GitHub repository tramgianguyen/phofrauddetect, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tramgianguyen/phofrauddetect
- Repository: https://github.com/tramgianguyen/phofrauddetect

## Overview

##### Project type and purpose
- This is an end-to-end machine-learning pipeline for detecting Vietnamese scam dialogues using a pre-trained PhoBERT model.
- Key use case: ingest raw scam/legitimate conversation data, expand and augment it via AI (Groq API) and NLP techniques, preprocess for tokenization, train a transformer-based classifier, then serve it for inference.

##### File structure
- Top-level Python scripts implement each pipeline stage:
- Generate_data.py & regenerate_failed_data.py – data-expansion via external Groq API
- Data_Augmentation.py – back-translation, synonym replacement, noise injection
- preprocessing.py – tokenization (PhoBERT tokenizer), sliding-window chunking, train/val/test split
- training-phobert.ipynb – model fine-tuning with Hugging Face Trainer
- inference_phobert.py & test-token.py – runtime inference and tokenization checks
- Data artifacts and models are stored in:
- Raw CSVs: scam_dataset.csv, expanded_scam_all_types.csv, augmented_expanded_scam_dataset.csv
- preprocessed_scam_dataset/{train,validation,test}
- model/ (config.json, model.safetensors, training_args.bin)
- Configuration and dependencies:
- requirements.txt
- README.md outlines hardware, software, and usage

##### README for additional insights
- Describes 6-step process: data generation, augmentation, preprocessing, training, token check, inference
- Highlights external Groq API usage, PhoBERT tokenizer/model (vinai/phobert-large), and performance targets (>90% accuracy, F1)
- Notes checkpointing, batch/chunk processing for memory optimization

a. Main components
1. Raw Data Storage
- scam_dataset.csv
2. Data Generation Service
- Generate_data.py + regenerate_failed_data.py
- External Groq API
3. Data Augmentation Module
- Data_Augmentation.py
4. Preprocessing Module
- preprocessing.py
- PhoBERT tokenizer
5. Model Training
- training-phobert.ipynb
- Hugging Face Trainer, PyTorch
6. Model Artifacts Storage
- model/ directory
7. Inference Service
- inference_phobert.py, test-token.py
- CLI flags for --text, --file, --add_markers
8. Evaluation & Monitoring (optional)
- logs, metrics (accuracy, F1)

b. Relationships and data flows

c. Architectural patterns and design principles
- Modular pipeline – each stage as an isolated script/module
- Checkpointing in augmentation to ensure fault tolerance
- Batch/chunk processing for memory efficiency
- Separation of concerns: data generation vs. augmentation vs. preprocessing vs. training vs. serving

d. Key technologies/frameworks
- Python 3.8+
- Groq API (optional)
- Transformers (Hugging Face) + PhoBERT (vinai/phobert-large)
- PyTorch 2.3.1, accelerate, datasets
- pandas, numpy, scikit-learn
- sentencepiece, googletrans

- Blue – Storage (CSV files, model artifacts)
- Orange – Processing modules (scripts, notebook)
- Green – External services (Groq API, GPU compute)
- Rectangles for compute modules

## Diagram (Mermaid)

```mermaid
graph TB
    %% Storage Nodes
    RawData["Raw Data\nscam_dataset.csv"]:::storage
    ExpandedData["Expanded Dataset\nexpanded_scam_all_types.csv"]:::storage
    AugmentedCSV["Augmented Dataset\naugmented_expanded_scam_dataset.csv"]:::storage
    TokenStore["Tokenized Splits\npreprocessed_scam_dataset/\n(train, validation, test)"]:::storage
    ModelArt["Model Artifacts\nmodel/"]:::storage
    Prediction["Prediction Output"]:::storage

    %% Processing Modules
    Generate["Generate_data.py"]:::process
    Regenerate["regenerate_failed_data.py"]:::process
    DataAug["Data Augmentation Module\nData_Augmentation.py"]:::process
    Preprocess["Preprocessing Module\npreprocessing.py"]:::process
    Training["Model Training\ntraining-phobert.ipynb"]:::process
    Inference1["Inference Service\ninference_phobert.py\n--text, --file, --add_markers"]:::process
    Inference2["Token Check\ntest-token.py"]:::process

    %% Config & Docs
    Requirements["requirements.txt"]:::config
    Readme["README.md"]:::config

    %% External Services
    Groq["Groq API"]:::external
    GPU["GPU / Compute Resource"]:::external

    %% Data Flow
    RawData --> Generate
    RawData --> Regenerate
    Generate --> ExpandedData
    Regenerate --> ExpandedData
    ExpandedData --> DataAug
    DataAug --> AugmentedCSV
    AugmentedCSV --> Preprocess
    Preprocess --> TokenStore
    TokenStore --> Training
    Training --> ModelArt
    ModelArt --> Inference1
    ModelArt --> Inference2
    Inference1 --> Prediction
    Inference2 --> Prediction

    %% Optional / External Interactions
    DataAug -. Checkpoints .-> DataAug
    Training ---|uses GPU| GPU

    %% Click Events
    click RawData "https://github.com/tramgianguyen/phofrauddetect/blob/main/scam_dataset.csv"
    click Generate "https://github.com/tramgianguyen/phofrauddetect/blob/main/Generate_data.py"
    click Regenerate "https://github.com/tramgianguyen/phofrauddetect/blob/main/regenerate_failed_data.py"
    click DataAug "https://github.com/tramgianguyen/phofrauddetect/blob/main/Data_Augmentation.py"
    click Preprocess "https://github.com/tramgianguyen/phofrauddetect/blob/main/preprocessing.py"
    click Training "https://github.com/tramgianguyen/phofrauddetect/blob/main/training-phobert.ipynb"
    click Inference1 "https://github.com/tramgianguyen/phofrauddetect/blob/main/inference_phobert.py"
    click Inference2 "https://github.com/tramgianguyen/phofrauddetect/blob/main/test-token.py"
    click Requirements "https://github.com/tramgianguyen/phofrauddetect/blob/main/requirements.txt"
    click Readme "https://github.com/tramgianguyen/phofrauddetect/blob/main/README.md"
    click ExpandedData "https://github.com/tramgianguyen/phofrauddetect/blob/main/expanded_scam_all_types.csv"
    click AugmentedCSV "https://github.com/tramgianguyen/phofrauddetect/blob/main/augmented_expanded_scam_dataset.csv"

    %% Styles
    classDef storage fill:#D6EAF8,stroke:#2980B9,stroke-width:2px;
    classDef process fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef external fill:#D5F5E3,stroke:#27AE60,stroke-width:2px,stroke-dasharray: 5 5;
    classDef config fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
```
