How wesroots/msc_project_refactor works
This is a biomedical knowledge-graph construction pipeline: it searches PubMed for abstracts, extracts entities and relations with an LLM, and converts extraction output into evaluation-ready predictions. Entity normalization is implemented separately; evaluation, relation validation, and graph construction are named in the pipeline but their modules are empty or the orchestration is unfinished. Edges reflect shown calls, reads, and API requests; the separate normalization module is not shown wired into the main script.
Main entry points: extraction.py LLM extractor · kg_pipeline.py Pipeline runner · corpus.py Corpus builder · processing.py Output processor
Components 18
Corpus collection
- Search queries
configs/queries.py - Corpus builder
src/pipeline/corpus.py
LLM extraction
- Path configuration
configs/paths.py - Prompt template
data/prompt_template.txt - Extraction guidelines
data/br_ext_guidelines.txt - Few-shot examples
data/fs_block.txt - LLM extractor
src/pipeline/extraction.py
Entity normalization
- Entity normalizer
src/pipeline/entity_normalisation.py
Processing and KG
- Output processor
src/pipeline/processing.py - Extraction log
data/run/extraction_log.json - Evaluation
src/pipeline/evaluation.py - Relation validation
src/pipeline/relation_validation.py - Graph construction
src/pipeline/graph.py
Other components
- Pipeline runner
scripts/kg_pipeline.py - PubMed
- OpenAI API
- NCBI E-utilities
- Cellosaurus API
Connections 14
- Pipeline runner → Corpus builder: calls
scripts/kg_pipeline.py - Pipeline runner → LLM extractor: calls
scripts/kg_pipeline.py - Pipeline runner → Output processor: calls
scripts/kg_pipeline.py - Corpus builder → Search queries: imports queries
src/pipeline/corpus.py - Corpus builder → PubMed: searches and fetches
src/pipeline/corpus.py - LLM extractor → OpenAI API: requests extraction
src/pipeline/extraction.py - LLM extractor → Path configuration: loads paths from
src/pipeline/extraction.py - LLM extractor → Prompt template: loads prompt
src/pipeline/extraction.py - LLM extractor → Extraction guidelines: loads guidelines
src/pipeline/extraction.py - LLM extractor → Few-shot examples: loads examples
src/pipeline/extraction.py - Entity normalizer → NCBI E-utilities: searches identifiers
src/pipeline/entity_normalisation.py - Entity normalizer → Cellosaurus API: searches cell lines
src/pipeline/entity_normalisation.py - Output processor → Path configuration: imports output path
src/pipeline/processing.py - Output processor → Extraction log: reads outputs
src/pipeline/processing.py