How gpl-ebih/applying-rag-to-enhance-vietnamese-poetry-generation-with-correct-structures works
Project type and overall purpose
• This is a machine-learning driven application that incorporates Retrieval-Augmented Generation (RAG) techniques to generate Vietnamese poetry. It includes data preprocessing, model training (using LSTM and Transformer models), and a Flask-based web interface to deliver the results.
• The core functionality revolves around combining ML-based poem generation with a retrieval module (using Faiss) to incorporate existing Vietnamese poems for context and structure.
File structure to identify major components
• Backend / Model and RAG Infrastructure:
• Files like app.py, pythonfile.py, solver.py, and config.pt indicate the backend logic.
• The pytorch_model.pt file suggests a pre-trained or intermediate machine learning model.
• The poem-generative.ipynb notebook is used for experimentation or demonstrating the generation process.
• The requirements.txt and Dockerfile show that the system is containerized and built using a set of Python and ML dependencies.
• Static & Templates – Web Interface:
• The static directory (with CSS, JS, fonts, images, and SASS files) alongside the templates folder (with at least an index.html) are clearly related to the Flask web interface.
• Data & Dependencies:
• The vocab.pkl, configuration files (.env, config.pt), and the dataset (implicitly referenced through README and used in the RAG system) show the dependency on external data for building and training the ML models.
• Deployment and Environment:
• The Dockerfile and .env indicate that the application is set up with environment-specific configurations and containerization for ease of deployment.
Key architectural components
a. Data Sources and Preprocessing:
• A component that represents the Vietnamese poems dataset (mentioned in the README) which feeds into the data processing and model training pipelines.
b. Model Training & Inference:
• An ML module that covers the training phases for LSTM and Transformer models.
• Use of PyTorch and scikit-learn libraries and the role of the pytorch_model.pt.
c. RAG Component:
• A retrieval system that uses Faiss for indexing and querying the poem dataset. This component should interact with the generation model to provide context and structure enhancement.
d. API Integrations:
• External API modules or calls to services from OpenAI, NVIDIA, and Meta, which are integrated to augment the generation process. Show their connection to the core ML and RAG components.
e. Web Interface (Flask Application):
• The Flask app (in app.py) that serves the front-end content using templates, static assets, and routes for user interaction.
f. Deployment & Containerization:
• Data flow from the dataset (poems & vocab) to preprocessing and then feeding into the ML model training pipeline.
• External API calls (OpenAI, NVIDIA, Meta) to the generation module as augmentation inputs.
• Flask application component as the user-facing layer that collects requests, triggers the ML/RAG process, and then returns a generated poem.
Architectural Patterns and Design Principles
• Modularity: Separate components for data processing, ML model training/inference, RAG retrieval, and the web interface.
• Separation of concerns, ensuring that the frontend (static assets and templates) is separate from the ML/RAG backend.
• Use clear labels and directional flows to indicate how data moves from one component to the next.