# nemesgyadam/blackmirrors02e01 architecture

> Architecture diagram of the GitHub repository nemesgyadam/blackmirrors02e01, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nemesgyadam/blackmirrors02e01
- Repository: https://github.com/nemesgyadam/blackmirrors02e01

## Overview

##### Project Type & Purpose
- The project is a machine learning (ML) pipeline that fine-tunes a GPT-3.5 model to mimic a user's Facebook conversation style.
- It involves data ingestion (downloading and storing Facebook chat history), data transformation (parsing and formatting), training data preparation (conversion into a JSONL dataset), and finally, utilizing OpenAI’s fine-tuning API/CLI and chat generation via Jupyter notebooks.

##### Analyzing the File Structure
- Top-Level Files:
- .gitignore, ReadMe.md, requirements.txt – standard configuration and dependency management.
- chat.ipynb and parser.ipynb – two Jupyter Notebooks that serve different purposes (chat generation vs. data parsing).
- results_EN.txt & results_HUN.txt – likely outputs/results from the model interactions showing testing or performance in different language settings.
- Directories:
- source_conversations: Storage for raw chat history data as downloaded from Facebook. This folder is organized by subject and conversation partners, with JSON files representing individual messages.
- training_data: Stores processed data (in JSONL format) prepared for fine-tuning the GPT-3.5 model.

##### Architectural Components
- Data Source (Facebook Chat Data): The raw data is collected manually by downloading the Facebook chat history.
- Data Ingestion & Storage: The downloaded data is stored in the source_conversations directory which preserves the hierarchical conversation structure.
- Data Parsing Component:
- Implemented in parser.ipynb, this notebook’s role is to transform the raw Facebook download into the GPT-formatted data.
- It reads data from source_conversations, then outputs a processed dataset (formatted and stored in training_data).
- Training Data Component: training_data contains the processed and formatted JSONL files required for model fine-tuning.
- Fine-Tuning Process:
- This part of the system interacts with OpenAI’s API or CLI using commands detailed in the README.
- It leverages the processed training data to fine-tune the GPT-3.5 model.
- Chat Generation Component:
- Implemented in chat.ipynb, this Jupyter Notebook is used to generate answer messages.
- It likely interacts (or after the fine-tuning process, utilizes the newly trained model) to produce responses that mimic the user’s conversation style.

##### System Interactions & Data Flow
- The parser retrieves and transforms these raw files → outputs a new JSONL dataset saved in training_data.
- Fine-tuning is performed by either invoking the OpenAI API (via CLI or online platform) against the JSONL file from training_data.
- Once the model is fine-tuned, chat.ipynb is used to generate responses using the trained model.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        A["Facebook Data Source (Manual Download)"]
        B["Source Data Storage"]
        D["Training Data Store"]
    end

    %% Processing & Training Layer
    subgraph "Processing & Training Layer"
        C["Data Parsing Module"]
        E["Fine-Tuning Process"]
    end

    %% Interaction Layer
    subgraph "Interaction Layer"
        F["Chat Generation Module"]
    end

    %% Data Flow
    A -->|"downloads to"| B
    B -->|"parsed by"| C
    C -->|"outputs to"| D
    D -->|"used for"| E
    E -->|"enables"| F

    %% Click Events
    click B "https://github.com/nemesgyadam/blackmirrors02e01/tree/master/source_conversations"
    click C "https://github.com/nemesgyadam/blackmirrors02e01/blob/master/parser.ipynb"
    click D "https://github.com/nemesgyadam/blackmirrors02e01/tree/master/training_data"
    click E "https://github.com/nemesgyadam/blackmirrors02e01/blob/master/ReadMe.md"
    click F "https://github.com/nemesgyadam/blackmirrors02e01/blob/master/chat.ipynb"

    %% Styles
    classDef manual fill:#fee,stroke:#333,stroke-width:2px;
    classDef dataLayer fill:#f9f,stroke:#333,stroke-width:2px;
    classDef processing fill:#bbf,stroke:#333,stroke-width:2px;
    classDef external fill:#cfc,stroke:#333,stroke-width:2px;

    class A manual;
    class B,D dataLayer;
    class C,E processing;
    class F external;
```
