How tramgianguyen/phofrauddetect works
Project type and purpose
• This is an end-to-end machine-learning pipeline for detecting Vietnamese scam dialogues using a pre-trained PhoBERT model.
• Key use case: ingest raw scam/legitimate conversation data, expand and augment it via AI (Groq API) and NLP techniques, preprocess for tokenization, train a transformer-based classifier, then serve it for inference.
File structure
• Top-level Python scripts implement each pipeline stage:
• Generate_data.py & regenerate_failed_data.py – data-expansion via external Groq API
• Data_Augmentation.py – back-translation, synonym replacement, noise injection
• preprocessing.py – tokenization (PhoBERT tokenizer), sliding-window chunking, train/val/test split
• training-phobert.ipynb – model fine-tuning with Hugging Face Trainer
• inference_phobert.py & test-token.py – runtime inference and tokenization checks
• Data artifacts and models are stored in:
• Raw CSVs: scam_dataset.csv, expanded_scam_all_types.csv, augmented_expanded_scam_dataset.csv
• preprocessed_scam_dataset/{train,validation,test}
• model/ (config.json, model.safetensors, training_args.bin)
• Configuration and dependencies:
• requirements.txt
• README.md outlines hardware, software, and usage
README for additional insights
• Describes 6-step process: data generation, augmentation, preprocessing, training, token check, inference
• Highlights external Groq API usage, PhoBERT tokenizer/model (vinai/phobert-large), and performance targets (>90% accuracy, F1)
• Notes checkpointing, batch/chunk processing for memory optimization
a. Main components
1. Raw Data Storage
• scam_dataset.csv
2. Data Generation Service
• Generate_data.py + regenerate_failed_data.py
• External Groq API
3. Data Augmentation Module
• Data_Augmentation.py
4. Preprocessing Module
• preprocessing.py
• PhoBERT tokenizer
5. Model Training
• training-phobert.ipynb
• Hugging Face Trainer, PyTorch
6. Model Artifacts Storage
• model/ directory
7. Inference Service
• inference_phobert.py, test-token.py
• CLI flags for --text, --file, --add_markers
8. Evaluation & Monitoring (optional)
• logs, metrics (accuracy, F1)
b. Relationships and data flows
c. Architectural patterns and design principles
• Modular pipeline – each stage as an isolated script/module
• Checkpointing in augmentation to ensure fault tolerance
• Batch/chunk processing for memory efficiency
• Separation of concerns: data generation vs. augmentation vs. preprocessing vs. training vs. serving
d. Key technologies/frameworks
• Python 3.8+
• Groq API (optional)
• Transformers (Hugging Face) + PhoBERT (vinai/phobert-large)
• PyTorch 2.3.1, accelerate, datasets
• pandas, numpy, scikit-learn
• sentencepiece, googletrans
• Blue – Storage (CSV files, model artifacts)
• Orange – Processing modules (scripts, notebook)
• Green – External services (Groq API, GPU compute)
• Rectangles for compute modules