How marcodsn/academic-chains works
Project type and purpose
• This is an open-source data-pipeline and evaluation toolkit whose goal is to build, verify, and package a high-quality reasoning-and-intuition chains dataset from open-access research papers.
• It is *not* a web app or microservice architecture, but rather a modular Python-script-driven pipeline that orchestrates data ingestion, LLM-based generation, automatic filtering, multi-LLM verification, and dataset packaging.
File structure
• Top-level directories:
• data/ – stores intermediate pipeline outputs (JSONL, checkpoints, figures)
• prompts/ – few-shot prompt files and example papers for LLMs
• scripts/ – the core pipeline code, subdivided into data_generation, data_processing, plus utilities
• src/ – evaluation and (planned) training modules
• models/ – placeholders or artifacts for fine-tuned models
• Configuration and orchestration:
• requirements.txt lists Python dependencies (LLM SDKs, HF datasets, etc.)
• shell scripts (generate_reqs.sh, upload_to_hf.py) for environment setup and final publishing
• Data flow style suggests a directed acyclic pipeline (DAG) rather than a server-based architecture.
README for additional insights
• Pipeline stages are clearly described: metadata gathering, PDF text extraction, chain generation, automated filtering, LLM-based verification, final packaging on HuggingFace.
• External dependencies: arXiv API, arxiv-markdown extractor, multiple LLM providers (Cohere, Gemini, Ollama, TogetherAI), HF Datasets.
Main components
a. Data Ingestion & Extraction
• Component: “Metadata Fetcher” (scripts/data_generation/togetherai.py + arXiv API)
• Component: “PDF Markdown Extractor” (arxiv-markdown pipeline invoked by data_generation scripts)
b. Chain Generation Stage
• Multiple “Curator” modules (curator_cohere.py, curator_gemini.py, curator_ollama.py, curator_togetherai.py)
• Input: extracted markdown + prompts/example_papers
• Output: raw reasoning chains in JSONL under data/jsonls/
c. Automated Filtering
• scripts/data_processing/process.py to remove formatting errors, off-topic entries
• scripts/data_processing/deduplicate.py to remove duplicates
d. LLM-Based Verification
• scripts/data_processing/verify_dataset.py and merge_verifiers.py
• External calls to verifier models (Qwen3, Mistral, Gemma) via API
• Aggregation into verifier_results and suitability_score
e. Packaging & Publishing
• scripts/upload_to_hf.py uses HuggingFace Datasets SDK to push final JSONL
• data/checkpoints/ used for intermediate saves
f. Evaluation & Training (planned)
• src/eval/inference.py for scoring models on held-out data
• src/train/train.py and merge_lora.py for fine-tuning with LoRA
Relationships & data flow
1. arXiv API → Metadata Fetcher → PDF Extractor
2. PDF Extractor + prompts/ → Chain Generators (four parallel modules) → raw JSONL
3. raw JSONL → Filtering scripts → cleaned JSONL
4. cleaned JSONL → Verifier scripts (parallel calls to multiple LLMs) → merged verifier results
5. merged JSONL → upload_to_hf → HuggingFace Dataset
6. final dataset → src/eval → metrics; optionally → src/train → fine-tuned models
• External services as distinct nodes: arXiv API, LLM Providers, HuggingFace Hub.
Architectural patterns & design principles
• Pipeline (DAG) pattern: modular, each stage is an independent script
• Pluggable LLM interfaces: each curator_*.py is an adapter for one provider
• Data storage as JSONL: append-only, checkpointable
• QC via ensemble verification: multi-agent consensus pattern