How klaudia-nazarko/nlg-text-generation works
Main Components and Their Roles
• Data Ingestion & Preprocessing:
• Web Scraper: Represented by the web_scraper.ipynb notebook, which gathers and possibly cleans the raw fairy tale data.
• Data Utilities: Files like Text.py and functions.py likely contain helper functions for text cleaning, tokenization, and preparation.
• Data Splitting: The train_test_split.ipynb notebook suggests a step dedicated to splitting the corpus into training and testing sets.
• Model Implementations:
• Markov Chain Model: Developed in Chain_class.py and demonstrated in markov_chain.ipynb. This model uses a stochastic, memory-less approach based solely on previous words.
• LSTM Neural Network: Implemented in LSTM_class.py and showcased in lstm.ipynb. This component handles sequences with memory to predict the next word.
• GPT-2 Transformer: Shown in gpt_2.ipynb. This notebook introduces transformers with self-attention mechanisms for advanced text generation.
• Result Generation & Evaluation:
• Each model (Markov Chain, LSTM, GPT-2) is tied to its specific notebook where the generation and evaluation (and possibly fine-tuning or training) happens.
Relationships and Interactions
• A preprocessing layer (Text.py and functions.py) will transform and prepare the raw data.
• The processed data flows into three parallel model pipelines – each pipeline representing one method (Markov Chain, LSTM, or GPT-2).
• Optional: A train/test splitting component feeding into those pipelines since model training and evaluation are key phases.
• Finally, the output (generated text) can be shown as a common endpoint where the user can view and compare results from the different approaches.
Architectural Patterns/Design Elements
• Modular Approach: Even though the project file tree is flat (without subdirectories), the separation of concerns is clear. Each file and notebook plays a specific role (data extraction, model implementation, utility functions).
• Pipeline Structure: The architecture follows a sequential pipeline pattern: Data Extraction → Preprocessing → Model Training/Inference → Output.
• Experimentation Notebooks: Using Jupyter notebooks for different model implementations illustrates an experimental and research-focused design rather than a production API style system.
Technologies, Frameworks, and Libraries
• Although not explicitly listed in the tree, the notebooks imply dependencies on deep learning frameworks (likely TensorFlow or PyTorch for LSTM and GPT-2) and standard Python libraries for NLP.
• Markov Chain logic is implemented directly in Python without heavy external libraries.