# marcodsn/academic-chains architecture

> Architecture diagram of the GitHub repository marcodsn/academic-chains, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marcodsn/academic-chains
- Repository: https://github.com/marcodsn/academic-chains

## Overview

##### Project type and purpose
- This is an open-source data-pipeline and evaluation toolkit whose goal is to build, verify, and package a high-quality reasoning-and-intuition chains dataset from open-access research papers.
- It is *not* a web app or microservice architecture, but rather a modular Python-script-driven pipeline that orchestrates data ingestion, LLM-based generation, automatic filtering, multi-LLM verification, and dataset packaging.

##### File structure
- Top-level directories:
- data/           – stores intermediate pipeline outputs (JSONL, checkpoints, figures)
- prompts/        – few-shot prompt files and example papers for LLMs
- scripts/        – the core pipeline code, subdivided into data_generation, data_processing, plus utilities
- src/            – evaluation and (planned) training modules
- models/         – placeholders or artifacts for fine-tuned models
- Configuration and orchestration:
- requirements.txt lists Python dependencies (LLM SDKs, HF datasets, etc.)
- shell scripts (generate_reqs.sh, upload_to_hf.py) for environment setup and final publishing
- Data flow style suggests a directed acyclic pipeline (DAG) rather than a server-based architecture.

##### README for additional insights
- Pipeline stages are clearly described: metadata gathering, PDF text extraction, chain generation, automated filtering, LLM-based verification, final packaging on HuggingFace.
- External dependencies: arXiv API, arxiv-markdown extractor, multiple LLM providers (Cohere, Gemini, Ollama, TogetherAI), HF Datasets.

##### Main components
a. Data Ingestion & Extraction
- Component: “Metadata Fetcher” (scripts/data_generation/togetherai.py + arXiv API)
- Component: “PDF Markdown Extractor” (arxiv-markdown pipeline invoked by data_generation scripts)
b. Chain Generation Stage
- Multiple “Curator” modules (curator_cohere.py, curator_gemini.py, curator_ollama.py, curator_togetherai.py)
- Input: extracted markdown + prompts/example_papers
- Output: raw reasoning chains in JSONL under data/jsonls/
c. Automated Filtering
- scripts/data_processing/process.py to remove formatting errors, off-topic entries
- scripts/data_processing/deduplicate.py to remove duplicates
d. LLM-Based Verification
- scripts/data_processing/verify_dataset.py and merge_verifiers.py
- External calls to verifier models (Qwen3, Mistral, Gemma) via API
- Aggregation into `verifier_results` and `suitability_score`
e. Packaging & Publishing
- scripts/upload_to_hf.py uses HuggingFace Datasets SDK to push final JSONL
- data/checkpoints/ used for intermediate saves
f. Evaluation & Training (planned)
- src/eval/inference.py for scoring models on held-out data
- src/train/train.py and merge_lora.py for fine-tuning with LoRA

##### Relationships & data flow
1. arXiv API → Metadata Fetcher → PDF Extractor
2. PDF Extractor + prompts/ → Chain Generators (four parallel modules) → raw JSONL
3. raw JSONL → Filtering scripts → cleaned JSONL
4. cleaned JSONL → Verifier scripts (parallel calls to multiple LLMs) → merged verifier results
5. merged JSONL → upload_to_hf → HuggingFace Dataset
6. final dataset → src/eval → metrics; optionally → src/train → fine-tuned models
- External services as distinct nodes: arXiv API, LLM Providers, HuggingFace Hub.

##### Architectural patterns & design principles
- Pipeline (DAG) pattern: modular, each stage is an independent script
- Pluggable LLM interfaces: each curator_*.py is an adapter for one provider
- Data storage as JSONL: append-only, checkpointable
- QC via ensemble verification: multi-agent consensus pattern

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    ext_Arxiv["arXiv API"]:::external
    ext_LLM["LLM Providers"]:::external
    ext_HF["HuggingFace Hub"]:::external

    %% Storage
    data["data/ (Intermediate & Final Data Storage)"]:::storage
    models["models/ (Model Artifacts & Checkpoints)"]:::storage
    prompts["prompts/ (Prompt Definitions & Examples)"]:::storage

    %% Data Ingestion & Extraction
    subgraph "Data Ingestion & Extraction"
        direction TB
        MF["togetherai.py: Metadata Fetcher"]:::stage
        PDF["PDF Markdown Extractor"]:::stage
    end

    %% Chain Generation
    subgraph "Chain Generation"
        direction TB
        CC["curator_cohere.py: Cohere Curator"]:::stage
        CG["curator_gemini.py: Gemini Curator"]:::stage
        CO["curator_ollama.py: Ollama Curator"]:::stage
        CT["curator_togetherai.py: TogetherAI Curator"]:::stage
    end

    %% Automated Filtering
    subgraph "Automated Filtering"
        direction TB
        PROC["process.py: Format & Topic Filtering"]:::stage
        DEDUP["deduplicate.py: Deduplication"]:::stage
    end

    %% LLM-Based Verification
    subgraph "LLM-Based Verification"
        direction TB
        VER["verify_dataset.py: Per-example Verification"]:::stage
        MERGEV["merge_verifiers.py: Aggregation"]:::stage
    end

    %% Packaging & Publishing
    subgraph "Packaging & Publishing"
        direction TB
        UPLOAD["upload_to_hf.py: HF Upload"]:::stage
    end

    %% Evaluation & Training
    subgraph "Evaluation & Training"
        direction TB
        EVAL["inference.py: Evaluation & Metrics"]:::stage
        TRAIN["train.py: Fine-tuning Pipeline"]:::stage
        MERGE_LO["merge_lora.py: LoRA Merge Utility"]:::stage
    end

    %% Data Flow
    ext_Arxiv --> MF
    MF --> PDF
    PDF --> CC
    PDF --> CG
    PDF --> CO
    PDF --> CT

    prompts --> CC
    prompts --> CG
    prompts --> CO
    prompts --> CT

    ext_LLM --> CC
    ext_LLM --> CG
    ext_LLM --> CO
    ext_LLM --> CT

    CC --> data
    CG --> data
    CO --> data
    CT --> data

    data --> PROC
    PROC --> DEDUP
    DEDUP --> data

    data --> VER
    ext_LLM --> VER
    VER --> MERGEV
    MERGEV --> data

    data --> UPLOAD
    UPLOAD --> ext_HF

    data --> EVAL
    data --> TRAIN
    TRAIN --> MERGE_LO
    MERGE_LO --> models

    %% Click Events
    click MF "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_generation/togetherai.py"
    click CC "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_generation/curator_cohere.py"
    click CG "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_generation/curator_gemini.py"
    click CO "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_generation/curator_ollama.py"
    click CT "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_generation/curator_togetherai.py"
    click PROC "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_processing/process.py"
    click DEDUP "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_processing/deduplicate.py"
    click VER "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_processing/verify_dataset.py"
    click MERGEV "https://github.com/marcodsn/academic-chains/blob/main/scripts/data_processing/merge_verifiers.py"
    click UPLOAD "https://github.com/marcodsn/academic-chains/blob/main/scripts/upload_to_hf.py"
    click data "https://github.com/marcodsn/academic-chains/tree/main/data/"
    click models "https://github.com/marcodsn/academic-chains/tree/main/models/"
    click prompts "https://github.com/marcodsn/academic-chains/tree/main/prompts/"
    click EVAL "https://github.com/marcodsn/academic-chains/blob/main/src/eval/inference.py"
    click TRAIN "https://github.com/marcodsn/academic-chains/blob/main/src/train/train.py"
    click MERGE_LO "https://github.com/marcodsn/academic-chains/blob/main/src/train/merge_lora.py"

    %% Styles
    classDef storage fill:#ddeeff,stroke:#000080,stroke-width:1px
    classDef stage fill:#ccffcc,stroke:#006400,stroke-width:1px
    classDef external fill:#ffe0b2,stroke:#ff8c00,stroke-width:1px
```
