# tnagar72/tanay-s-nd-zettelkasten architecture

> Architecture diagram of the GitHub repository tnagar72/tanay-s-nd-zettelkasten, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tnagar72/tanay-s-nd-zettelkasten
- Repository: https://github.com/tnagar72/tanay-s-nd-zettelkasten

## Overview

##### Main Components

a) Data Processing Pipeline:
- Input Sources
- CommonSenseQA dataset (indicated by CommonSenseQA_train_questions.txt)
- Hinglish questions (hinglish_questions.json, hinglish_questions.txt)
- Logic questions (Logic.json, logicQ.txt)

b) Text Processing Components:
- Text Synthesis Module (indicated by Chat-GPT CS Text Synthesis directory)
- Data Transformation Tools (prettify_print scripts)
- Format Conversion Tools (parquet_to_csv.py)

c) Evaluation System:
- Benchmark Testing Module
- Performance Comparison Framework
- Language Model Integration Points

**[Data Layer]**
- ↓
**[Processing Layer]**
- ↓
**[Model Layer]**
- ↓
**[Evaluation Layer]**

##### Key Elements to Highlight

a) Data Flow:

b) Processing Stages:
- Text Synthesis Stage (Chat-GPT integration)
- Data Transformation Stage
- Model Fine-tuning Stage
- Evaluation Stage

##### Technical Details

- File formats used (JSON, TXT, JSONL)
- Integration points with external systems (like OpenAI/Chat-GPT)
- Data validation and preprocessing steps

##### Component Relationships

- Data sources and preprocessing scripts
- Text synthesis and data transformation
- Model training and evaluation
- Benchmark testing and results analysis

##### Notes Section

A small notes section indicating:
- Current project phase
- Key dependencies
- Critical paths in the pipeline

This system design should help communicate both the technical architecture and the research methodology effectively to other team members and stakeholders.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Data_Layer
        CS[CommonSenseQA Dataset]:::data
        HQ[Hinglish Questions]:::data
        LQ[Logic Questions]:::data
    end

    subgraph Processing_Layer
        TS[Text Synthesis Module]:::process
        DT[Data Transformation Tools]:::process
        FC[Format Conversion Tool]:::process
        GPT[GPT Integration]:::external
    end

    subgraph Model_Layer
        FT[Fine-tuning Process]:::model
        CS_Gen[Code-Switched Data Generation]:::model
    end

    subgraph Evaluation_Layer
        BT[Benchmark Testing]:::eval
        PC[Performance Comparison]:::eval
        Results[Evaluation Results]:::eval
    end

    subgraph Documentation
        Doc[Project Documentation]:::docs
    end

    %% Data Flow
    CS --> TS
    HQ --> TS
    LQ --> TS
    TS --> DT
    DT --> FC
    GPT --> TS
    FC --> FT
    DT --> CS_Gen
    CS_Gen --> FT
    FT --> BT
    BT --> PC
    PC --> Results
    Results -.-> FT
    Doc -.-> BT

    %% Click Events
    click CS "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/Chat-GPT CS Text Synthesis/CommonSenseQA_train_questions.txt"
    click HQ "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/hinglish_questions.json"
    click LQ "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/Week4code/Logic.json"
    click TS "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/Chat-GPT CS Text Synthesis/All_Questions_through_GPT.py"
    click DT "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/Chat-GPT CS Text Synthesis/prettify_print-All_CS.py"
    click FC "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/Week4code/parquet_to_csv.py"
    click BT "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/Benchmarks.md"
    click GPT "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/Chat-GPT CS Text Synthesis/openai-test.py"
    click Doc "https://github.com/tnagar72/Tanay-s-ND-Zettelkasten/blob/main/README.md"

    %% Styling
    classDef data fill:#a2d2ff,stroke:#333,stroke-width:2px
    classDef process fill:#bde0fe,stroke:#333,stroke-width:2px
    classDef model fill:#ffafcc,stroke:#333,stroke-width:2px
    classDef eval fill:#ffc8dd,stroke:#333,stroke-width:2px
    classDef external fill:#cdb4db,stroke:#333,stroke-width:2px
    classDef docs fill:#fff1e6,stroke:#333,stroke-width:2px

    %% Legend
    subgraph Legend
        L1[Data Components]:::data
        L2[Processing Components]:::process
        L3[Model Components]:::model
        L4[Evaluation Components]:::eval
        L5[External Services]:::external
        L6[Documentation]:::docs
    end
```
