# noviscl/ai-researcher architecture

> Architecture diagram of the GitHub repository noviscl/ai-researcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noviscl/ai-researcher
- Repository: https://github.com/noviscl/ai-researcher

## Overview

##### Project type and purpose
- This is not a web or mobile app but a research‐ideation pipeline built in Python, orchestrated by shell scripts.
- Purpose: Given a research topic, the system retrieves related papers, generates ideas grounded in those papers, deduplicates and refines ideas, expands them into detailed project proposals, ranks and filters proposals, and outputs ranked, high‐quality research proposals.

##### File structure
- Top level:
- ai_researcher/ – main Python package
- prompts/ – prompt templates (.txt, .json, .py utils)
- scripts/ – bash wrappers to run modules end to end or individually
- src/ – Python modules implementing each stage of the pipeline
- figures/ – illustrative assets (for docs)
- results/ – stored outputs and statistics scripts
- requirements.txt – Python dependencies
- Each src file corresponds to one pipeline step or utility: e.g., lit_review.py, grounded_idea_gen.py, dedup_ideas.py, project_proposal_gen.py, project_proposal_ranking.py, project_proposal_filter.py, plus scoring, analysis, and utility modules.

##### README for additional insights
- The pipeline has six sequential modules:
1. Related Paper Search
2. Grounded Idea Generation
3. Idea Deduplication
4. Project Proposal Generation
5. Project Proposal Ranking
6. Project Proposal Filtering
- Each can be run standalone or as part of an end‐to‐end flow via scripts/end_to_end.sh
- External services: OpenAI, Semantic Scholar API, Anthropic API, sentence‐transformers embedding models

##### Elements
- Clear labels on each component: “Related Paper Search,” “Grounded Idea Gen,” etc.
- Blue for core modules
- Green for data stores/cache
- Orange for external services
- Gray for orchestration scripts

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Orchestration Layer"
        Exec["execute.py"]:::orchestration
        E2E["end_to_end.sh"]:::orchestration
    end

    subgraph "Prompt Repository"
        Prompts["prompts/"]:::datastore
    end

    subgraph "Pipeline Modules"
        M1["Related Paper Search"]:::module
        M2["Grounded Idea Generation"]:::module
        M3["Idea Deduplication"]:::module
        M4["Project Proposal Generation"]:::module
        M5["Project Proposal Ranking"]:::module
        M6["Project Proposal Filtering"]:::module
    end

    subgraph "Data Stores"
        DS1["Paper Cache"]:::datastore
        DS2["Idea Bank"]:::datastore
        DS3["Deduped Ideas"]:::datastore
        DS4["Proposals"]:::datastore
        DS5["Scored Proposals"]:::datastore
        Results["Final Results"]:::datastore
    end

    subgraph "External Services"
        ES1["Semantic Scholar API"]:::external
        ES2["OpenAI / Anthropic API"]:::external
        ES3["Embedding Service"]:::external
    end

    subgraph "Config"
        Keys["keys.json"]:::config
    end

    E2E --> M1
    M1 --> DS1
    M1 --> ES1
    M1 --> M2
    M2 --> DS1
    M2 --> Prompts
    M2 --> ES2
    M2 --> DS2
    M2 --> M3
    M3 --> DS2
    M3 --> ES3
    M3 --> DS3
    M3 --> M4
    M4 --> DS3
    M4 --> Prompts
    M4 --> ES2
    M4 --> DS4
    M4 --> M5
    M5 --> DS4
    M5 --> Prompts
    M5 --> ES2
    M5 --> DS5
    M5 --> M6
    M6 --> DS5
    M6 --> DS1
    M6 --> Prompts
    M6 --> ES2
    M6 --> Results

    Exec --> E2E

    click Exec "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/execute.py"
    click E2E "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/scripts/end_to_end.sh"
    click Prompts "https://github.com/noviscl/ai-researcher/tree/main/ai_researcher/prompts/"
    click M1 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/scripts/lit_review.sh"
    click M1 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/lit_review.py"
    click M1 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/lit_review_tools.py"
    click M1 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/format_from_pdf.py"
    click M2 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/scripts/grounded_idea_gen.sh"
    click M2 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/grounded_idea_gen.py"
    click M3 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/scripts/dedup_ideas.sh"
    click M3 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/dedup_ideas.py"
    click M3 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/analyze_ideas_semantic_similarity.py"
    click M3 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/analyze_ideas_token_similarity.py"
    click M4 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/scripts/project_proposal_gen.sh"
    click M4 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/experiment_plan_gen.py"
    click M5 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/scripts/project_proposal_ranking.sh"
    click M5 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/score_ranking.py"
    click M5 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/binary_ranking.py"
    click M5 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/excitement_ranking.py"
    click M5 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/tournament_ranking.py"
    click M5 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/idea_direct_score.py"
    click M6 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/scripts/project_proposal_filter.sh"
    click M6 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/filter_ideas.py"
    click M6 "https://github.com/noviscl/ai-researcher/blob/main/ai_researcher/src/feasibility_check.py"
    click Results "https://github.com/noviscl/ai-researcher/tree/main/results/"
    click Results "https://github.com/noviscl/ai-researcher/blob/main/results/data_points_all_anonymized.json"
    click Results "https://github.com/noviscl/ai-researcher/blob/main/results/stats_overall.py"
    click Results "https://github.com/noviscl/ai-researcher/blob/main/results/stats_per_idea.py"
    click Results "https://github.com/noviscl/ai-researcher/blob/main/results/stats_per_reviewer.py"
    click Results "https://github.com/noviscl/ai-researcher/blob/main/results/stats_per_topic.py"

    classDef orchestration fill:#cccccc,stroke:#333,stroke-width:1px;
    classDef module fill:#bbdefb,stroke:#1976d2,stroke-width:1px;
    classDef datastore fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px;
    classDef external fill:#ffe0b2,stroke:#ef6c00,stroke-width:1px;
    classDef config fill:#e0e0e0,stroke:#424242,stroke-width:1px;
```
