# uci-soe/fairytaleqadata architecture

> Architecture diagram of the GitHub repository uci-soe/fairytaleqadata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uci-soe/fairytaleqadata
- Repository: https://github.com/uci-soe/fairytaleqadata

## Overview

##### Project type and purpose
- This is an **open-source dataset repository** (“FairytaleQA”) designed for NLP researchers and educators.
- It provides a rich collection of CSV files (stories, QA-pairs, sentences) plus helper code to load, split, and preprocess data, and a Huggingface integration for easy consumption.

##### File structure
- Top-level:
- data-by-origin/
- data-by-train-split/
- img/ (charts)
- starter.py (data loader entrypoint)
- story_meta.csv
- Under each `data-by-*` directory there are three parallel subtrees:
- questions/ (expert-labeled QA pairs)
- section-stories/ (paragraph-level splits)
- sentence-stories/ (sentence-level splits, via spaCy)
- Each split namespace (origin vs. train/test/val) repeats the same structure.
- Configuration: README.md describes usage; no database or web server; purely file-based.

##### README insights
- Explains two splitting strategies: by origin and by train/val/test.
- Mentions `starter.py` for retrieving and aggregating data using pandas.
- References spaCy’s `en_core_web_sm` for sentence segmentation.
- Mentions Huggingface Dataset loading.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Raw Data Storage"
        RawOrigin["Origin-based CSVs"]:::rawdata
        RawSplit["Train/Val/Test CSVs"]:::rawdata
        Meta["story_meta.csv"]:::rawdata
    end

    DataLoader["Data Loader (starter.py)"]:::processing

    subgraph "Preprocessing Pipelines"
        SectionO["Section Pipeline (Origin)"]:::processing
        SectionT["Section Pipeline (TrainSplit)"]:::processing
        SentenceO["Sentence Pipeline (Origin)"]:::processing
        SentenceT["Sentence Pipeline (TrainSplit)"]:::processing
        Aggregated["Aggregated DataFrames"]:::processing
    end

    API["Huggingface Dataset API"]:::api

    subgraph "Documentation & Assets"
        Doc["README.md"]:::docs
        Img["Charts (img/)"]:::docs
    end

    Users["End Users / Researchers"]:::consumer

    RawOrigin -->|ingest| DataLoader
    RawSplit -->|ingest| DataLoader
    Meta -->|lookup| DataLoader

    DataLoader -->|process| SectionO
    DataLoader -->|process| SectionT
    DataLoader -->|process| SentenceO
    DataLoader -->|process| SentenceT

    SectionO -->|outputs| Aggregated
    SectionT -->|outputs| Aggregated
    SentenceO -->|outputs| Aggregated
    SentenceT -->|outputs| Aggregated

    Aggregated -->|export| API
    API -->|serve| Users

    Doc -.-> DataLoader
    Img -.-> DataLoader

    click RawOrigin "https://github.com/uci-soe/fairytaleqadata/tree/main/data-by-origin/"
    click RawSplit "https://github.com/uci-soe/fairytaleqadata/tree/main/data-by-train-split/"
    click Meta "https://github.com/uci-soe/fairytaleqadata/blob/main/story_meta.csv"
    click DataLoader "https://github.com/uci-soe/fairytaleqadata/blob/main/starter.py"
    click SectionO "https://github.com/uci-soe/fairytaleqadata/tree/main/data-by-origin/section-stories/"
    click SectionT "https://github.com/uci-soe/fairytaleqadata/tree/main/data-by-train-split/section-stories/"
    click SentenceO "https://github.com/uci-soe/fairytaleqadata/tree/main/data-by-origin/sentence-stories/"
    click SentenceT "https://github.com/uci-soe/fairytaleqadata/tree/main/data-by-train-split/sentence-stories/"
    click API "https://github.com/uci-soe/fairytaleqadata/blob/main/starter.py"
    click Doc "https://github.com/uci-soe/fairytaleqadata/blob/main/README.md"
    click Img "https://github.com/uci-soe/fairytaleqadata/tree/main/img/"

    classDef rawdata fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:1px
    classDef api fill:#FFA500,stroke:#333,stroke-width:1px
    classDef consumer fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef docs fill:#ADD8E6,stroke:#333,stroke-width:1px
```
