How tnagar72/tanay-s-nd-zettelkasten works
Main Components
a) Data Processing Pipeline:
• Input Sources
• CommonSenseQA dataset (indicated by CommonSenseQA_train_questions.txt)
• Hinglish questions (hinglish_questions.json, hinglish_questions.txt)
• Logic questions (Logic.json, logicQ.txt)
b) Text Processing Components:
• Text Synthesis Module (indicated by Chat-GPT CS Text Synthesis directory)
• Data Transformation Tools (prettify_print scripts)
• Format Conversion Tools (parquet_to_csv.py)
c) Evaluation System:
• Benchmark Testing Module
• Performance Comparison Framework
• Language Model Integration Points
[Data Layer]
• ↓
[Processing Layer]
• ↓
[Model Layer]
• ↓
[Evaluation Layer]
Key Elements to Highlight
a) Data Flow:
b) Processing Stages:
• Text Synthesis Stage (Chat-GPT integration)
• Data Transformation Stage
• Model Fine-tuning Stage
• Evaluation Stage
Technical Details
• File formats used (JSON, TXT, JSONL)
• Integration points with external systems (like OpenAI/Chat-GPT)
• Data validation and preprocessing steps
Component Relationships
• Data sources and preprocessing scripts
• Text synthesis and data transformation
• Model training and evaluation
• Benchmark testing and results analysis
Notes Section
A small notes section indicating:
• Current project phase
• Key dependencies
• Critical paths in the pipeline
This system design should help communicate both the technical architecture and the research methodology effectively to other team members and stakeholders.