How moyiliyi/magellan-novelty-generation works
Project Type and Purpose
• This is an open-source research framework, not a user-facing web app. It provides a pipelined toolset for:
a) building a FAISS vector database from scientific paper abstracts,
b) running guided Monte Carlo Tree Search (Magellan) and baseline experiments using Qwen LLM weights,
c) analyzing and scoring the results with “LLM as Judge” scripts.
Top-Level Components (based on file_tree and README)
a. Data Storage & Preparation (“database/”)
• build_database.py: ingest raw JSON paper data → FAISS index + metadata + vectors.npy
• construct_test_dataset.py: sample a test set of thematic prompts for evaluation
b. Experiment Suite (“experiments/”)
• Multiple runner scripts: run_experiments_magellan.py, run_experiments_react.py, etc.
• Each script: loads FAISS DB, loads model weights, runs a generation policy, writes results JSON
c. Core Algorithm Library (“magellan/”)
• cg_mcts_qwen.py and cg_mcts_qwen_noter.py implement the Guided MCTS core
d. Analysis & Evaluation (“analysis/”)
• llm_judge.py: scores generated ideas via an LLM prompt
• analyze_results.py: aggregates scores into metrics
e. External Dependencies
• Qwen model weights (GPU or CPU via faiss-gpu/ faiss-cpu)
• Python environment (requirements.txt)
Relationships and Data Flow
1. Raw Data (./database/raw_data/*.json)
↓ build_database.py
2. FAISS Vector DB + Metadata (./database/vector_db/)
↓ construct_test_dataset.py
3. Test Themes (./data/test_set/)
↓ experiments/run_experiments_*.py
4. Results JSON (./results/*.json)
↓ analysis/llm_judge.py
5. Judgment Scores
↓ analysis/analyze_results.py
6. Final Metrics & Plots
Architectural Patterns & Design Principles
• Modular Pipeline: clear separation of stages (DB prep, experiments, analysis)
• Extensible Experiment Interface: each runner script shares CLI args (--modelpath, --dbpath, etc.)
• Core Algorithm Library: detached from orchestration, living in magellan/ for reuse or extension