How daverlon/engtoid works
Main Components
• Data Ingestion & Dataset Preparation:
• Components for dataset download and initial ingestion (see src/download_dataset.py).
• Data cleaning and preprocessing (src/clean_dataset.py and src/utils/preprocessing.py).
• Dataset encoding and tokenization (src/encode_dataset.py, src/generate_vocab.py, and src/dataset/ENGtoID.py).
• Configuration & Experiment Management:
• A configuration module (src/config.py) which centrally manages key parameters.
• Notebooks (src/ICT303 - Assignment 2.ipynb) as interactive interfaces to manage experiments.
• Model Development:
• Model definition files in src/models (model.py, seq2seq_lstm.py).
• LSTM seq2seq architecture used in translation.
• Training & Evaluation:
• Training scripts (src/train_model.py, src/train_seq2seq_lstm.py).
• Trainer implementations in src/trainers (trainer.py).
• Test scripts (src/test_dataset.py, src/test_preprocessing.py, src/test_trainer.py, src/test_valid.py) indicating unit or integration tests.
• Utility & Helper Functions:
• Utility functions (src/utils/coder.py, src/utils/metrics.py) that support the processing and evaluation, ensuring data flows and performance metrics are managed.
Out Relationships & Data Flow
• Dataset download feeding into cleaning scripts.
• Pass the cleaned data into encoding/tokenization.
• Generate vocabulary as a standalone step.
• Follow with the “Model Development” block:
• How the preprocessed, tokenized data enters the model training module.
• Training scripts with the trainer component that orchestrates the model’s learning process.
• Link “Configuration” as a central module that influences both data processing and the training modules.
• “Utility” functions as cross-cutting modules that provide support (e.g., metrics for evaluation, preprocessing wrappers).
Architectural Patterns & Frameworks
• Separation of concerns: Data preprocessing, model definition, training, and utility functions are all modularized.
• The project leverages a typical machine learning pipeline pattern: ingest data → preprocess → define model → train → evaluate.
• Although not explicitly a microservices architecture, the clear separation into modules (data, model, trainer, utilities) represents a loosely coupled design ideal for iterative experiments.
Additional Details
• Even though deployment isn’t explicitly mentioned, the setup script (src/setup.sh) hints at an automated setup process for the environment.