# di37/ner-electrical-engineering-dataset architecture

> Architecture diagram of the GitHub repository di37/ner-electrical-engineering-dataset, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/di37/ner-electrical-engineering-dataset
- Repository: https://github.com/di37/ner-electrical-engineering-dataset

## Overview

##### Project Components
- Dataset Generation Module:
- Represented by the script “01_dataset_creation.py” in the dataset_creation_pipeline directory.
- This module sends structured prompts (defined in the prompt folder) to an LLM (e.g., OpenAI) to generate raw annotated data.
- It is responsible for interacting with the LLM through asynchronous API calls.
- CSV Data Output:
- The generated data is saved as CSV files (intermediate output).
- A storage (or temporary data store) component that holds these raw CSV files.
- Data Conversion Module:
- Represented by “02_csvs_to_hf_dataset.ipynb”
- This notebook reads the CSV files and converts them into a Hugging Face “DatasetDict” format.
- Dataset Upload Module:
- Represented by “03_upload_to_hf_hub.ipynb”
- This module leverages the Hugging Face API for uploading the processed dataset along with metadata and license details.
- Configuration & Environment Management:
- Configuration for environment variables (e.g., HF_TOKEN and OPENAI_API_KEY stored in a .env file).
- Supporting Components:
- The “prompt” folder contains the logic and structured prompt definitions which feed into the dataset creation process.
- Placeholder directories (“electrical_engineering_ner_dataset” and “ner_dataset”) can be noted as repositories or placeholders for final dataset storage or future expansion.

##### Relationships and Interactions
- From the “Prompt” component (prompt folder) to the “Dataset Generation Module” (01_dataset_creation.py) as the source of structured queries.
- From the dataset generation module to “CSV Data Output” as the module saves raw data.

##### Architectural Patterns and Design Principles
- Pipeline (sequential) nature of this system:
- The multi-stage process (data generation, processing, and uploading) should be shown as distinct stages.
- Asynchronous processing within the dataset generation stage, indicating efficient handling of batch requests.
- Configuration management (environment variables, API keys) as an underpinning layer accessible by the modules that require external service connections.

##### Relevant Technologies, Frameworks, and Libraries
- Python scripts and Jupyter notebooks are used for processing.
- External libraries such as Hugging Face’s datasets library and API integrations for LLM calls.
- Mention that the project follows good modular practices by separating each key functionality into individual components (scripts, notebooks, and prompt management).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input and Configuration Layer
    subgraph "Input & Config"
        config["Configuration (.env)"]:::config
        promptGen["Prompt Generation"]:::internal
    end

    %% Data Generation Stage
    subgraph "Data Generation Stage"
        ds["Dataset Creation"]:::internal
    end

    %% Intermediate Storage
    subgraph "Intermediate Storage"
        csvS["((CSV Storage/Files))"]:::storage
    end

    %% Data Conversion Stage
    subgraph "Data Conversion Stage"
        dc["Dataset Conversion"]:::internal
    end

    %% Data Upload Stage
    subgraph "Data Upload Stage"
        du["Dataset Upload"]:::internal
    end

    %% Final Storage Repositories
    subgraph "Final Storage Repositories"
        ee["Final Dataset Storage (EE)"]:::storage
        ner["Final Dataset Storage (NER)"]:::storage
    end

    %% External Services
    oa["OpenAI API"]:::external
    hf["Hugging Face Hub"]:::external

    %% Connections
    config -->|"feeds"| promptGen
    promptGen -->|"provides prompts"| ds
    ds -->|"writes CSV"| csvS
    csvS -->|"reads CSV"| dc
    dc -->|"converts data"| du
    du -->|"uploads dataset"| hf
    du -->|"stores dataset"| ee
    du -->|"stores dataset"| ner
    ds -.->|"async call"| oa

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef external fill:#FFD700,stroke:#000,stroke-width:1px;
    classDef config fill:#D3D3D3,stroke:#000,stroke-width:1px;
    classDef storage fill:#90EE90,stroke:#000,stroke-width:1px;

    %% Click Events
    click promptGen "https://github.com/di37/ner-electrical-engineering-dataset/blob/main/prompt/prompt.py"
    click ds "https://github.com/di37/ner-electrical-engineering-dataset/blob/main/dataset_creation_pipeline/01_dataset_creation.py"
    click dc "https://github.com/di37/ner-electrical-engineering-dataset/blob/main/dataset_creation_pipeline/02_csvs_to_hf_dataset.ipynb"
    click du "https://github.com/di37/ner-electrical-engineering-dataset/blob/main/dataset_creation_pipeline/03_upload_to_hf_hub.ipynb"
    click ee "https://github.com/di37/ner-electrical-engineering-dataset/tree/main/electrical_engineering_ner_dataset"
    click ner "https://github.com/di37/ner-electrical-engineering-dataset/tree/main/ner_dataset"
```
