# moyiliyi/magellan-novelty-generation architecture

> Architecture diagram of the GitHub repository moyiliyi/magellan-novelty-generation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/moyiliyi/magellan-novelty-generation
- Repository: https://github.com/moyiliyi/magellan-novelty-generation

## Overview

##### Project Type and Purpose
- This is an open-source research framework, not a user-facing web app. It provides a pipelined toolset for:
a) building a FAISS vector database from scientific paper abstracts,
b) running guided Monte Carlo Tree Search (Magellan) and baseline experiments using Qwen LLM weights,
c) analyzing and scoring the results with “LLM as Judge” scripts.

##### Top-Level Components (based on file_tree and README)
a. Data Storage & Preparation (“database/”)
- build_database.py: ingest raw JSON paper data → FAISS index + metadata + vectors.npy
- construct_test_dataset.py: sample a test set of thematic prompts for evaluation
b. Experiment Suite (“experiments/”)
- Multiple runner scripts: run_experiments_magellan.py, run_experiments_react.py, etc.
- Each script: loads FAISS DB, loads model weights, runs a generation policy, writes results JSON
c. Core Algorithm Library (“magellan/”)
- cg_mcts_qwen.py and cg_mcts_qwen_noter.py implement the Guided MCTS core
d. Analysis & Evaluation (“analysis/”)
- llm_judge.py: scores generated ideas via an LLM prompt
- analyze_results.py: aggregates scores into metrics
e. External Dependencies
- Qwen model weights (GPU or CPU via faiss-gpu/ faiss-cpu)
- Python environment (requirements.txt)

##### Relationships and Data Flow
1. Raw Data (./database/raw_data/*.json)
↓ build_database.py
2. FAISS Vector DB + Metadata (./database/vector_db/)
↓ construct_test_dataset.py
3. Test Themes (./data/test_set/)
↓ experiments/run_experiments_*.py
4. Results JSON (./results/*.json)
↓ analysis/llm_judge.py
5. Judgment Scores
↓ analysis/analyze_results.py
6. Final Metrics & Plots

##### Architectural Patterns & Design Principles
- Modular Pipeline: clear separation of stages (DB prep, experiments, analysis)
- Extensible Experiment Interface: each runner script shares CLI args (`--modelpath`, `--dbpath`, etc.)
- Core Algorithm Library: detached from orchestration, living in magellan/ for reuse or extension

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Stage
    RawData["Raw Data JSON\n(database/raw_data/*.json)"]:::blue
    BuildDB["build_database.py"]:::blue
    FAISSDB["FAISS Vector DB\n(database/vector_db/)"]:::blue
    TestDataset["construct_test_dataset.py"]:::blue
    TestThemes["Test Themes JSON\n(data/test_set/)"]:::blue

    RawData -->|"ingest JSON"| BuildDB
    BuildDB -->|"faiss_index.bin + vectors.npy\n+ metadata"| FAISSDB
    FAISSDB -->|“sample themes”| TestDataset
    TestDataset -->|"test_set.json"| TestThemes

    %% Core Algorithm Library
    subgraph "Core Algorithm Library"
        direction TB
        MagCore1["cg_mcts_qwen.py"]:::green
        MagCore2["cg_mcts_qwen_noter.py"]:::green
    end

    %% Experimentation Stage
    subgraph "Experimentation"
        direction TB
        EXP_MAG["run_experiments_magellan.py"]:::green
        EXP_REACT["run_experiments_react.py"]:::green
        EXP_BASE["run_experiments_baseline.py"]:::green
        EXP_COT["run_experiments_cot.py"]:::green
        EXP_TOT["run_experiments_tot.py"]:::green
        EXP_AB1["run_ablation_no_guidance.py"]:::green
        EXP_AB2["run_ablation_no_novelty.py"]:::green
        EXP_AB3["run_ablation_no_progress.py"]:::green
        EXP_AI["run_aiscientist_generate_ideas.py"]:::green
        EXP_SCI["run_scipip_idea_expand.py"]:::green
        Results["Results JSON\n(results/*.json)"]:::green
    end

    TestThemes -->|"prompts"| EXP_MAG
    TestThemes --> EXP_REACT
    TestThemes --> EXP_BASE
    TestThemes --> EXP_COT
    TestThemes --> EXP_TOT
    TestThemes --> EXP_AB1
    TestThemes --> EXP_AB2
    TestThemes --> EXP_AB3
    TestThemes --> EXP_AI
    TestThemes --> EXP_SCI

    EXP_MAG --> MagCore1
    EXP_MAG --> MagCore2
    EXP_REACT --> MagCore1
    EXP_REACT --> MagCore2
    EXP_BASE --> MagCore1
    EXP_BASE --> MagCore2
    EXP_COT --> MagCore1
    EXP_COT --> MagCore2
    EXP_TOT --> MagCore1
    EXP_TOT --> MagCore2
    EXP_AB1 --> MagCore1
    EXP_AB2 --> MagCore1
    EXP_AB3 --> MagCore1
    EXP_AI --> MagCore1
    EXP_SCI --> MagCore1

    EXP_MAG --> Results
    EXP_REACT --> Results
    EXP_BASE --> Results
    EXP_COT --> Results
    EXP_TOT --> Results
    EXP_AB1 --> Results
    EXP_AB2 --> Results
    EXP_AB3 --> Results
    EXP_AI --> Results
    EXP_SCI --> Results

    %% Analysis Stage
    subgraph "Analysis"
        direction TB
        Judge["llm_judge.py"]:::orange
        Scores["Score Annotations"]:::orange
        Analyzer["analyze_results.py"]:::orange
        Metrics["Final Metrics & Plots"]:::orange
    end

    Results -->|“generated JSON”| Judge
    Judge -->|“scores.json”| Scores
    Scores -->|“aggregated”| Analyzer
    Analyzer --> Metrics

    %% External Dependencies
    QWEN["Qwen Model Weights"]:::gray
    FAISS_DEP["FAISS GPU/CPU"]:::gray
    REQ["requirements.txt"]:::gray

    FAISS_DEP -.-> BuildDB
    QWEN -.-> BuildDB
    QWEN -.-> EXP_MAG
    QWEN -.-> EXP_REACT
    QWEN -.-> EXP_BASE
    QWEN -.-> EXP_COT
    QWEN -.-> EXP_TOT
    QWEN -.-> EXP_AB1
    QWEN -.-> EXP_AB2
    QWEN -.-> EXP_AB3
    QWEN -.-> EXP_AI
    QWEN -.-> EXP_SCI
    REQ -.-> BuildDB
    REQ -.-> EXP_MAG
    REQ -.-> Judge

    %% Click Events
    click BuildDB "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/database/build_database.py"
    click TestDataset "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/database/construct_test_dataset.py"
    click EXP_MAG "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_experiments_magellan.py"
    click EXP_REACT "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_experiments_react.py"
    click EXP_BASE "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_experiments_baseline.py"
    click EXP_COT "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_experiments_cot.py"
    click EXP_TOT "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_experiments_tot.py"
    click EXP_AB1 "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_ablation_no_guidance.py"
    click EXP_AB2 "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_ablation_no_novelty.py"
    click EXP_AB3 "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_ablation_no_progress.py"
    click EXP_AI "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_aiscientist_generate_ideas.py"
    click EXP_SCI "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/experiments/run_scipip_idea_expand.py"
    click MagCore1 "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/magellan/cg_mcts_qwen.py"
    click MagCore2 "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/magellan/cg_mcts_qwen_noter.py"
    click Judge "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/analysis/llm_judge.py"
    click Analyzer "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/analysis/analyze_results.py"
    click REQ "https://github.com/moyiliyi/magellan-novelty-generation/blob/main/requirements.txt"

    %% Styles
    classDef blue fill:#cce5ff,stroke:#auto
    classDef green fill:#d4edda,stroke:#auto
    classDef orange fill:#ffe5b4,stroke:#auto
    classDef gray fill:#e2e3e5,stroke:#auto,stroke-dasharray: 5 5
```
