# simondedman/elasmo_analyses architecture

> Architecture diagram of the GitHub repository simondedman/elasmo_analyses, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/simondedman/elasmo_analyses
- Repository: https://github.com/simondedman/elasmo_analyses

## Overview

Shark Oracle builds a research corpus and turns it into searchable analyses: papers are acquired, made text-searchable, classified with evidence, enriched, and analyzed. Separate product branches cover conference-abstract ingestion, the Author Atlas, and a local natural-language RAG prototype. The README documents the main pipeline stages; source excerpts confirm selected calls and data-loading relationships. Other unsampled subsystems are represented only where the README or source index supports them.

Main entry points: [`scripts/conf_abstracts/run_pipeline.py`](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/run_pipeline.py) (Abstract pipeline), [`scripts/rag/query.py`](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/query.py) (RAG query), [`frontend/network_atlas/src/App.jsx`](https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/App.jsx) (Atlas application), [`scripts/acquire_cascade.py`](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/acquire_cascade.py) (Paper acquisition)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_corpus["Paper corpus"]
  node_acquire["Paper acquisition<br/>[acquire_cascade.py]"]
  node_ingest["PDF ingestion<br/>[ingest_pdfs.py]"]
  node_dedup["Duplicate detection<br/>[ingest_hook.py]"]
  node_ocr["PDF OCR<br/>[ocr_library.py]"]
  node_extract["Schema extraction"]
  node_enrich["Author and impact enrichment"]
  node_corpusdb[("Research corpus<br/>[papers_data_io.py]")]
end

subgraph group_abstracts["Conference abstracts"]
  node_abstractpipe["Abstract pipeline<br/>[run_pipeline.py]"]
  node_abstractparse["Abstract parsing<br/>[parse_eea.py]"]
  node_cdrom["CD-ROM ingestion"]
  node_xlsx["Spreadsheet ingestion<br/>[ingest_oa_xlsx.py]"]
  node_abstracttag["Abstract tagging<br/>[tag.py]"]
  node_abstractload[("Abstract storage<br/>[load.py]")]
  node_abstractconfig["Abstract configuration<br/>[config.py]"]
  node_abstractlexicon["Abstract lexicon<br/>[lexicon.py]"]
  node_abstractnames["Author normalization<br/>[names.py]"]
end

subgraph group_analysis["Analysis and atlas"]
  node_analyses["Corpus analysis"]
  node_atlasentry["Atlas application<br/>[App.jsx]"]
  node_atlasdata["Atlas data loaders<br/>[dataLoaders.js]"]
  node_filters["Atlas filters<br/>[FilterPanel.jsx]"]
  node_profile["Author profiles<br/>[ProfileCard.jsx]"]
end

subgraph group_rag["Corpus Q&amp;A"]
  node_ragquery["RAG query<br/>[query.py]"]
  node_retrieval["Filtered retrieval<br/>[retrieval.py]"]
  node_rerank["Result reranking<br/>[rerank.py]"]
  node_entailment["Evidence assessment<br/>[entailment.py]"]
  node_llm["Answer generation<br/>[llm_backend.py]"]
end

node_researcher(("Researcher"))
node_atlasuser(("Atlas visitor"))
node_raguser(("Questioner"))

node_researcher -->|"submits paper"| node_acquire
node_acquire -->|"finalizes via"| node_ingest
node_ingest -->|"checks duplicates"| node_dedup
node_acquire -->|"uses data I/O"| node_corpusdb
node_acquire -.->|"continues to OCR"| node_ocr
node_ocr -->|"supplies text"| node_extract
node_extract -->|"feeds enrichment"| node_enrich
node_enrich -->|"feeds analysis"| node_analyses
node_abstractpipe -->|"parses records"| node_abstractparse
node_abstractpipe -->|"tags records"| node_abstracttag
node_abstractpipe -->|"loads records"| node_abstractload
node_cdrom -->|"loads records"| node_abstractload
node_xlsx -.->|"loads records"| node_abstractload
node_abstractpipe -->|"uses settings"| node_abstractconfig
node_abstractpipe -.->|"uses vocabulary"| node_abstractlexicon
node_abstractpipe -.->|"normalizes authors"| node_abstractnames
node_atlasuser -->|"explores atlas"| node_atlasentry
node_atlasentry -->|"loads datasets"| node_atlasdata
node_atlasentry -.->|"renders controls"| node_filters
node_atlasentry -->|"renders profile"| node_profile
node_analyses -.->|"provides atlas data"| node_atlasentry
node_raguser -->|"asks question"| node_ragquery
node_ragquery -->|"retrieves candidates"| node_retrieval
node_ragquery -->|"reranks results"| node_rerank
node_ragquery -->|"assesses evidence"| node_entailment
node_ragquery -->|"generates answer"| node_llm

click node_acquire "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/acquire_cascade.py"
click node_ingest "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/ingest_pdfs.py"
click node_dedup "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/dedup/ingest_hook.py"
click node_ocr "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/ocr_library.py"
click node_extract "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/extract_schema_columns.py"
click node_enrich "https://github.com/simondedman/elasmo_analyses/tree/main/scripts"
click node_corpusdb "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/lib/papers_data_io.py"
click node_abstractpipe "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/run_pipeline.py"
click node_abstractparse "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/parse_eea.py"
click node_cdrom "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/ingest_jmih_cdrom.py"
click node_xlsx "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/ingest_oa_xlsx.py"
click node_abstracttag "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/tag.py"
click node_abstractload "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/load.py"
click node_abstractconfig "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/config.py"
click node_abstractlexicon "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/lexicon.py"
click node_abstractnames "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/names.py"
click node_analyses "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/build_analysis_tables.py"
click node_atlasentry "https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/App.jsx"
click node_atlasdata "https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/lib/dataLoaders.js"
click node_filters "https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/components/FilterPanel.jsx"
click node_profile "https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/components/ProfileCard.jsx"
click node_ragquery "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/query.py"
click node_retrieval "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/retrieval.py"
click node_rerank "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/rerank.py"
click node_entailment "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/entailment.py"
click node_llm "https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/llm_backend.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_acquire,node_ingest,node_dedup,node_ocr,node_extract,node_enrich,node_corpusdb toneBlue
class node_abstractpipe,node_abstractparse,node_cdrom,node_xlsx,node_abstracttag,node_abstractload,node_abstractconfig,node_abstractlexicon,node_abstractnames toneAmber
class node_analyses,node_atlasentry,node_atlasdata,node_filters,node_profile toneMint
class node_ragquery,node_retrieval,node_rerank,node_entailment,node_llm toneRose
class node_researcher,node_atlasuser,node_raguser toneIndigo
```

## Components

### Paper corpus

- [Paper acquisition](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/acquire_cascade.py): `scripts/acquire_cascade.py`
- [PDF ingestion](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/ingest_pdfs.py): `scripts/ingest_pdfs.py`
- [Duplicate detection](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/dedup/ingest_hook.py): `scripts/dedup/ingest_hook.py`
- [PDF OCR](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/ocr_library.py): `scripts/ocr_library.py`
- [Schema extraction](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/extract_schema_columns.py): `scripts/extract_schema_columns.py`
- [Author and impact enrichment](https://github.com/simondedman/elasmo_analyses/tree/main/scripts): `scripts`
- [Research corpus](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/lib/papers_data_io.py): `scripts/lib/papers_data_io.py`

### Conference abstracts

- [Abstract pipeline](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/run_pipeline.py): `scripts/conf_abstracts/run_pipeline.py`
- [Abstract parsing](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/parse_eea.py): `scripts/conf_abstracts/parse_eea.py`
- [CD-ROM ingestion](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/ingest_jmih_cdrom.py): `scripts/conf_abstracts/ingest_jmih_cdrom.py`
- [Spreadsheet ingestion](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/ingest_oa_xlsx.py): `scripts/conf_abstracts/ingest_oa_xlsx.py`
- [Abstract tagging](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/tag.py): `scripts/conf_abstracts/tag.py`
- [Abstract storage](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/load.py): `scripts/conf_abstracts/load.py`
- [Abstract configuration](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/config.py): `scripts/conf_abstracts/config.py`
- [Abstract lexicon](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/lexicon.py): `scripts/conf_abstracts/lexicon.py`
- [Author normalization](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/conf_abstracts/names.py): `scripts/conf_abstracts/names.py`

### Analysis and atlas

- [Corpus analysis](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/build_analysis_tables.py): `scripts/build_analysis_tables.py`
- [Atlas application](https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/App.jsx): `frontend/network_atlas/src/App.jsx`
- [Atlas data loaders](https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/lib/dataLoaders.js): `frontend/network_atlas/src/lib/dataLoaders.js`
- [Atlas filters](https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/components/FilterPanel.jsx): `frontend/network_atlas/src/components/FilterPanel.jsx`
- [Author profiles](https://github.com/simondedman/elasmo_analyses/blob/main/frontend/network_atlas/src/components/ProfileCard.jsx): `frontend/network_atlas/src/components/ProfileCard.jsx`

### Corpus Q&A

- [RAG query](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/query.py): `scripts/rag/query.py`
- [Filtered retrieval](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/retrieval.py): `scripts/rag/retrieval.py`
- [Result reranking](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/rerank.py): `scripts/rag/rerank.py`
- [Evidence assessment](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/entailment.py): `scripts/rag/entailment.py`
- [Answer generation](https://github.com/simondedman/elasmo_analyses/blob/main/scripts/rag/llm_backend.py): `scripts/rag/llm_backend.py`

### Other components

- **Researcher**
- **Atlas visitor**
- **Questioner**

## Connections

- Researcher → Paper acquisition: submits paper
- Paper acquisition → PDF ingestion: finalizes via (evidence: `scripts/acquire_cascade.py`)
- PDF ingestion → Duplicate detection: checks duplicates (evidence: `scripts/ingest_pdfs.py`)
- Paper acquisition → Research corpus: uses data I/O (evidence: `scripts/acquire_cascade.py`)
- Paper acquisition → PDF OCR: continues to OCR
- PDF OCR → Schema extraction: supplies text
- Schema extraction → Author and impact enrichment: feeds enrichment
- Author and impact enrichment → Corpus analysis: feeds analysis
- Abstract pipeline → Abstract parsing: parses records (evidence: `scripts/conf_abstracts/run_pipeline.py`)
- Abstract pipeline → Abstract tagging: tags records (evidence: `scripts/conf_abstracts/run_pipeline.py`)
- Abstract pipeline → Abstract storage: loads records (evidence: `scripts/conf_abstracts/run_pipeline.py`)
- CD-ROM ingestion → Abstract storage: loads records (evidence: `scripts/conf_abstracts/ingest_jmih_cdrom.py`)
- Spreadsheet ingestion → Abstract storage: loads records (evidence: `scripts/conf_abstracts/ingest_oa_xlsx.py`)
- Abstract pipeline → Abstract configuration: uses settings (evidence: `scripts/conf_abstracts/run_pipeline.py`)
- Abstract pipeline → Abstract lexicon: uses vocabulary
- Abstract pipeline → Author normalization: normalizes authors
- Atlas visitor → Atlas application: explores atlas
- Atlas application → Atlas data loaders: loads datasets (evidence: `frontend/network_atlas/src/App.jsx`)
- Atlas application → Atlas filters: renders controls (evidence: `frontend/network_atlas/src/App.jsx`)
- Atlas application → Author profiles: renders profile (evidence: `frontend/network_atlas/src/App.jsx`)
- Corpus analysis → Atlas application: provides atlas data
- Questioner → RAG query: asks question
- RAG query → Filtered retrieval: retrieves candidates (evidence: `scripts/rag/query.py`)
- RAG query → Result reranking: reranks results (evidence: `scripts/rag/query.py`)
- RAG query → Evidence assessment: assesses evidence (evidence: `scripts/rag/query.py`)
- RAG query → Answer generation: generates answer (evidence: `scripts/rag/query.py`)
