How simondedman/elasmo_analyses works
Shark Oracle builds a research corpus and turns it into searchable analyses: papers are acquired, made text-searchable, classified with evidence, enriched, and analyzed. Separate product branches cover conference-abstract ingestion, the Author Atlas, and a local natural-language RAG prototype. The README documents the main pipeline stages; source excerpts confirm selected calls and data-loading relationships. Other unsampled subsystems are represented only where the README or source index supports them.
Main entry points: run_pipeline.py Abstract pipeline · query.py RAG query · App.jsx Atlas application · acquire_cascade.py Paper acquisition
Components 29
Paper corpus
- Paper acquisition
scripts/acquire_cascade.py - PDF ingestion
scripts/ingest_pdfs.py - Duplicate detection
scripts/dedup/ingest_hook.py - PDF OCR
scripts/ocr_library.py - Schema extraction
scripts/extract_schema_columns.py - Author and impact enrichment
scripts - Research corpus
scripts/lib/papers_data_io.py
Conference abstracts
- Abstract pipeline
scripts/conf_abstracts/run_pipeline.py - Abstract parsing
scripts/conf_abstracts/parse_eea.py - CD-ROM ingestion
scripts/conf_abstracts/ingest_jmih_cdrom.py - Spreadsheet ingestion
scripts/conf_abstracts/ingest_oa_xlsx.py - Abstract tagging
scripts/conf_abstracts/tag.py - Abstract storage
scripts/conf_abstracts/load.py - Abstract configuration
scripts/conf_abstracts/config.py - Abstract lexicon
scripts/conf_abstracts/lexicon.py - Author normalization
scripts/conf_abstracts/names.py
Analysis and atlas
- Corpus analysis
scripts/build_analysis_tables.py - Atlas application
frontend/network_atlas/src/App.jsx - Atlas data loaders
frontend/network_atlas/src/lib/dataLoaders.js - Atlas filters
frontend/network_atlas/src/components/FilterPanel.jsx - Author profiles
frontend/network_atlas/src/components/ProfileCard.jsx
Corpus Q&A
- RAG query
scripts/rag/query.py - Filtered retrieval
scripts/rag/retrieval.py - Result reranking
scripts/rag/rerank.py - Evidence assessment
scripts/rag/entailment.py - Answer generation
scripts/rag/llm_backend.py
Other components
- Researcher
- Atlas visitor
- Questioner
Connections 26
- Researcher → Paper acquisition: submits paper
- Paper acquisition → PDF ingestion: finalizes via
scripts/acquire_cascade.py - PDF ingestion → Duplicate detection: checks duplicates
scripts/ingest_pdfs.py - Paper acquisition → Research corpus: uses data I/O
scripts/acquire_cascade.py - Paper acquisition → PDF OCR: continues to OCR
- PDF OCR → Schema extraction: supplies text
- Schema extraction → Author and impact enrichment: feeds enrichment
- Author and impact enrichment → Corpus analysis: feeds analysis
- Abstract pipeline → Abstract parsing: parses records
scripts/conf_abstracts/run_pipeline.py - Abstract pipeline → Abstract tagging: tags records
scripts/conf_abstracts/run_pipeline.py - Abstract pipeline → Abstract storage: loads records
scripts/conf_abstracts/run_pipeline.py - CD-ROM ingestion → Abstract storage: loads records
scripts/conf_abstracts/ingest_jmih_cdrom.py - Spreadsheet ingestion → Abstract storage: loads records
scripts/conf_abstracts/ingest_oa_xlsx.py - Abstract pipeline → Abstract configuration: uses settings
scripts/conf_abstracts/run_pipeline.py - Abstract pipeline → Abstract lexicon: uses vocabulary
- Abstract pipeline → Author normalization: normalizes authors
- Atlas visitor → Atlas application: explores atlas
- Atlas application → Atlas data loaders: loads datasets
frontend/network_atlas/src/App.jsx - Atlas application → Atlas filters: renders controls
frontend/network_atlas/src/App.jsx - Atlas application → Author profiles: renders profile
frontend/network_atlas/src/App.jsx - Corpus analysis → Atlas application: provides atlas data
- Questioner → RAG query: asks question
- RAG query → Filtered retrieval: retrieves candidates
scripts/rag/query.py - RAG query → Result reranking: reranks results
scripts/rag/query.py - RAG query → Evidence assessment: assesses evidence
scripts/rag/query.py - RAG query → Answer generation: generates answer
scripts/rag/query.py