How di37/ner-electrical-engineering-dataset works
Project Components
• Dataset Generation Module:
• Represented by the script “01_dataset_creation.py” in the dataset_creation_pipeline directory.
• This module sends structured prompts (defined in the prompt folder) to an LLM (e.g., OpenAI) to generate raw annotated data.
• It is responsible for interacting with the LLM through asynchronous API calls.
• CSV Data Output:
• The generated data is saved as CSV files (intermediate output).
• A storage (or temporary data store) component that holds these raw CSV files.
• Data Conversion Module:
• Represented by “02_csvs_to_hf_dataset.ipynb”
• This notebook reads the CSV files and converts them into a Hugging Face “DatasetDict” format.
• Dataset Upload Module:
• Represented by “03_upload_to_hf_hub.ipynb”
• This module leverages the Hugging Face API for uploading the processed dataset along with metadata and license details.
• Configuration & Environment Management:
• Configuration for environment variables (e.g., HF_TOKEN and OPENAI_API_KEY stored in a .env file).
• Supporting Components:
• The “prompt” folder contains the logic and structured prompt definitions which feed into the dataset creation process.
• Placeholder directories (“electrical_engineering_ner_dataset” and “ner_dataset”) can be noted as repositories or placeholders for final dataset storage or future expansion.
Relationships and Interactions
• From the “Prompt” component (prompt folder) to the “Dataset Generation Module” (01_dataset_creation.py) as the source of structured queries.
• From the dataset generation module to “CSV Data Output” as the module saves raw data.
Architectural Patterns and Design Principles
• Pipeline (sequential) nature of this system:
• The multi-stage process (data generation, processing, and uploading) should be shown as distinct stages.
• Asynchronous processing within the dataset generation stage, indicating efficient handling of batch requests.
• Configuration management (environment variables, API keys) as an underpinning layer accessible by the modules that require external service connections.
Relevant Technologies, Frameworks, and Libraries
• Python scripts and Jupyter notebooks are used for processing.
• External libraries such as Hugging Face’s datasets library and API integrations for LLM calls.
• Mention that the project follows good modular practices by separating each key functionality into individual components (scripts, notebooks, and prompt management).