How nemesgyadam/blackmirrors02e01 works
Project Type & Purpose
• The project is a machine learning (ML) pipeline that fine-tunes a GPT-3.5 model to mimic a user's Facebook conversation style.
• It involves data ingestion (downloading and storing Facebook chat history), data transformation (parsing and formatting), training data preparation (conversion into a JSONL dataset), and finally, utilizing OpenAI’s fine-tuning API/CLI and chat generation via Jupyter notebooks.
Analyzing the File Structure
• Top-Level Files:
• .gitignore, ReadMe.md, requirements.txt – standard configuration and dependency management.
• chat.ipynb and parser.ipynb – two Jupyter Notebooks that serve different purposes (chat generation vs. data parsing).
• results_EN.txt & results_HUN.txt – likely outputs/results from the model interactions showing testing or performance in different language settings.
• Directories:
• source_conversations: Storage for raw chat history data as downloaded from Facebook. This folder is organized by subject and conversation partners, with JSON files representing individual messages.
• training_data: Stores processed data (in JSONL format) prepared for fine-tuning the GPT-3.5 model.
Architectural Components
• Data Source (Facebook Chat Data): The raw data is collected manually by downloading the Facebook chat history.
• Data Ingestion & Storage: The downloaded data is stored in the source_conversations directory which preserves the hierarchical conversation structure.
• Data Parsing Component:
• Implemented in parser.ipynb, this notebook’s role is to transform the raw Facebook download into the GPT-formatted data.
• It reads data from source_conversations, then outputs a processed dataset (formatted and stored in training_data).
• Training Data Component: training_data contains the processed and formatted JSONL files required for model fine-tuning.
• Fine-Tuning Process:
• This part of the system interacts with OpenAI’s API or CLI using commands detailed in the README.
• It leverages the processed training data to fine-tune the GPT-3.5 model.
• Chat Generation Component:
• Implemented in chat.ipynb, this Jupyter Notebook is used to generate answer messages.
• It likely interacts (or after the fine-tuning process, utilizes the newly trained model) to produce responses that mimic the user’s conversation style.
System Interactions & Data Flow
• The parser retrieves and transforms these raw files → outputs a new JSONL dataset saved in training_data.
• Fine-tuning is performed by either invoking the OpenAI API (via CLI or online platform) against the JSONL file from training_data.
• Once the model is fine-tuned, chat.ipynb is used to generate responses using the trained model.