# lang-jiaqi/omnisource architecture

> Architecture diagram of the GitHub repository lang-jiaqi/omnisource, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lang-jiaqi/omnisource
- Repository: https://github.com/lang-jiaqi/omnisource

## Overview

OmniSource turns configurable research, startup, and developer-tool signals into curated, ranked reports, published locally or to GitHub Issues. The primary workflow is the track-driven collector-to-publisher pipeline; a separate CS paper distiller enriches and reviews research candidates. The graph preserves both workflows and feedback-based personalization. Some relationships are supported only by the README or SOURCE INDEX references; unsampled subsystem details and their internal call paths are not assumed.

Main entry points: [`omnisource/main.py`](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/main.py) (Pipeline orchestrator), [`omnisource/distiller/pipeline.py`](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/pipeline.py) (Paper distiller pipeline), [`omnisource/publishers`](https://github.com/lang-jiaqi/omnisource/tree/main/omnisource/publishers) (Report publishers), [`omnisource/agents/curator.py`](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/curator.py) (Curation and shortlist)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Report workflow"]
  node_cli["CLI<br/>[cli.py]"]
  node_config["Track configuration<br/>[config.py]"]
  node_main["Pipeline orchestrator<br/>[main.py]"]
  node_sources["Signal sources"]
  node_signals["Unified signals<br/>[models.py]"]
  node_enrich["Signal enrichment<br/>[enrich.py]"]
  node_dedup["Deduplication<br/>[dedup.py]"]
  node_editor["Report editor<br/>[editor.py]"]
  node_publishers["Report publishers"]
end

subgraph group_curation["Curation and ranking"]
  node_curator["Curation and shortlist<br/>[curator.py]"]
  node_quality["Quality filtering<br/>[quality.py]"]
  node_analyst["LLM analysis<br/>[analyst.py]"]
  node_review["Paper review pass<br/>[review_distill.py]"]
  node_ranking["Ranking scores<br/>[scoring.py]"]
  node_personalization["Feedback personalization<br/>[personalization.py]"]
end

subgraph group_research["Paper distiller"]
  node_distillpipeline["Paper distiller pipeline<br/>[pipeline.py]"]
  node_papersources["Paper candidate sources<br/>[sources.py]"]
  node_taxonomy["Research taxonomy<br/>[taxonomy.py]"]
  node_router["Taxonomy router<br/>[router.py]"]
  node_fulltext["Full-text signals<br/>[fulltext.py]"]
  node_expertreview["Expert reviewers<br/>[reviewer.py]"]
  node_llmreview["LLM paper reviewer<br/>[llm_reviewer.py]"]
  node_selection["Candidate selection<br/>[selection.py]"]
end

subgraph group_support["State and integrations"]
  node_feedback["Repository feedback import"]
  node_memory[("Signal memory<br/>[store.py]")]
  node_cache[("Fetch cache<br/>[cache.py]")]
  node_llm["LLM providers<br/>[base.py]"]
end

node_user(("Track operator"))
node_github["GitHub Issues"]
node_localreports["Markdown and JSONL reports"]

node_user -->|"runs"| node_cli
node_cli -.->|"starts"| node_main
node_main -->|"loads tracks"| node_config
node_main -->|"collects signals"| node_sources
node_sources -->|"normalizes"| node_signals
node_main -->|"enriches"| node_enrich
node_main -->|"deduplicates"| node_dedup
node_main -->|"shortlists"| node_curator
node_main -->|"filters"| node_quality
node_main -->|"reviews papers"| node_review
node_main -.->|"analyzes"| node_analyst
node_curator -->|"consults"| node_analyst
node_curator -->|"scores"| node_ranking
node_main -->|"personalizes"| node_personalization
node_main -->|"edits report"| node_editor
node_main -->|"publishes"| node_publishers
node_publishers -->|"creates issues"| node_github
node_publishers -->|"writes reports"| node_localreports
node_main -->|"uses memory"| node_memory
node_enrich -->|"uses cache"| node_cache
node_feedback -->|"supplies events"| node_personalization
node_github -.->|"provides issue feedback"| node_feedback
node_user -.->|"runs distiller"| node_distillpipeline
node_distillpipeline -->|"fetches candidates"| node_papersources
node_distillpipeline -->|"loads taxonomy"| node_taxonomy
node_distillpipeline -->|"routes candidates"| node_router
node_distillpipeline -->|"enriches text"| node_fulltext
node_distillpipeline -->|"reviews candidates"| node_expertreview
node_distillpipeline -->|"requests review"| node_llmreview
node_distillpipeline -->|"selects papers"| node_selection
node_llmreview -->|"calls provider"| node_llm

click node_cli "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/cli.py"
click node_config "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/config.py"
click node_main "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/main.py"
click node_sources "https://github.com/lang-jiaqi/omnisource/tree/main/omnisource/sources"
click node_signals "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/models.py"
click node_enrich "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/enrich.py"
click node_dedup "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/dedup.py"
click node_curator "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/curator.py"
click node_quality "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/quality.py"
click node_analyst "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/analyst.py"
click node_review "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/review_distill.py"
click node_ranking "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/ranking/scoring.py"
click node_personalization "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/personalization.py"
click node_editor "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/editor.py"
click node_publishers "https://github.com/lang-jiaqi/omnisource/tree/main/omnisource/publishers"
click node_feedback "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/repository_feedback.py"
click node_memory "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/memory/store.py"
click node_cache "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/memory/cache.py"
click node_distillpipeline "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/pipeline.py"
click node_papersources "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/sources.py"
click node_taxonomy "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/taxonomy.py"
click node_router "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/router.py"
click node_fulltext "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/fulltext.py"
click node_expertreview "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/reviewer.py"
click node_llmreview "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/llm_reviewer.py"
click node_selection "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/selection.py"
click node_llm "https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/llm/base.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_main,node_sources,node_signals,node_enrich,node_dedup,node_editor,node_publishers toneBlue
class node_curator,node_quality,node_analyst,node_review,node_ranking,node_personalization toneAmber
class node_distillpipeline,node_papersources,node_taxonomy,node_router,node_fulltext,node_expertreview,node_llmreview,node_selection toneMint
class node_feedback,node_memory,node_cache,node_llm toneRose
class node_user,node_github,node_localreports toneIndigo
```

## Components

### Report workflow

- [CLI](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/cli.py): `omnisource/cli.py`
- [Track configuration](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/config.py): `omnisource/config.py`
- [Pipeline orchestrator](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/main.py): `omnisource/main.py`
- [Signal sources](https://github.com/lang-jiaqi/omnisource/tree/main/omnisource/sources): `omnisource/sources`
- [Unified signals](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/models.py): `omnisource/models.py`
- [Signal enrichment](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/enrich.py): `omnisource/enrich.py`
- [Deduplication](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/dedup.py): `omnisource/dedup.py`
- [Report editor](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/editor.py): `omnisource/agents/editor.py`
- [Report publishers](https://github.com/lang-jiaqi/omnisource/tree/main/omnisource/publishers): `omnisource/publishers`

### Curation and ranking

- [Curation and shortlist](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/curator.py): `omnisource/agents/curator.py`
- [Quality filtering](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/quality.py): `omnisource/agents/quality.py`
- [LLM analysis](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/analyst.py): `omnisource/agents/analyst.py`
- [Paper review pass](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/agents/review_distill.py): `omnisource/agents/review_distill.py`
- [Ranking scores](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/ranking/scoring.py): `omnisource/ranking/scoring.py`
- [Feedback personalization](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/personalization.py): `omnisource/personalization.py`

### Paper distiller

- [Paper distiller pipeline](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/pipeline.py): `omnisource/distiller/pipeline.py`
- [Paper candidate sources](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/sources.py): `omnisource/distiller/sources.py`
- [Research taxonomy](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/taxonomy.py): `omnisource/distiller/taxonomy.py`
- [Taxonomy router](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/router.py): `omnisource/distiller/router.py`
- [Full-text signals](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/fulltext.py): `omnisource/distiller/fulltext.py`
- [Expert reviewers](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/reviewer.py): `omnisource/distiller/reviewer.py`
- [LLM paper reviewer](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/llm_reviewer.py): `omnisource/distiller/llm_reviewer.py`
- [Candidate selection](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/distiller/selection.py): `omnisource/distiller/selection.py`

### State and integrations

- [Repository feedback import](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/repository_feedback.py): `omnisource/repository_feedback.py`
- [Signal memory](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/memory/store.py): `omnisource/memory/store.py`
- [Fetch cache](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/memory/cache.py): `omnisource/memory/cache.py`
- [LLM providers](https://github.com/lang-jiaqi/omnisource/blob/main/omnisource/llm/base.py): `omnisource/llm/base.py`

### Other components

- **Track operator**
- **GitHub Issues**
- **Markdown and JSONL reports**

## Connections

- Track operator → CLI: runs (evidence: `README.md`)
- CLI → Pipeline orchestrator: starts
- Pipeline orchestrator → Track configuration: loads tracks (evidence: `omnisource/main.py`)
- Pipeline orchestrator → Signal sources: collects signals (evidence: `omnisource/main.py`)
- Signal sources → Unified signals: normalizes (evidence: `omnisource/models.py`)
- Pipeline orchestrator → Signal enrichment: enriches (evidence: `omnisource/main.py`)
- Pipeline orchestrator → Deduplication: deduplicates
- Pipeline orchestrator → Curation and shortlist: shortlists (evidence: `omnisource/main.py`)
- Pipeline orchestrator → Quality filtering: filters (evidence: `omnisource/main.py`)
- Pipeline orchestrator → Paper review pass: reviews papers (evidence: `omnisource/main.py`)
- Pipeline orchestrator → LLM analysis: analyzes
- Curation and shortlist → LLM analysis: consults (evidence: `omnisource/agents/curator.py`)
- Curation and shortlist → Ranking scores: scores (evidence: `omnisource/agents/curator.py`)
- Pipeline orchestrator → Feedback personalization: personalizes (evidence: `omnisource/main.py`)
- Pipeline orchestrator → Report editor: edits report (evidence: `omnisource/main.py`)
- Pipeline orchestrator → Report publishers: publishes (evidence: `omnisource/main.py`)
- Report publishers → GitHub Issues: creates issues (evidence: `README.md`)
- Report publishers → Markdown and JSONL reports: writes reports (evidence: `README.md`)
- Pipeline orchestrator → Signal memory: uses memory (evidence: `omnisource/main.py`)
- Signal enrichment → Fetch cache: uses cache (evidence: `omnisource/enrich.py`)
- Repository feedback import → Feedback personalization: supplies events (evidence: `omnisource/personalization.py`)
- GitHub Issues → Repository feedback import: provides issue feedback
- Track operator → Paper distiller pipeline: runs distiller
- Paper distiller pipeline → Paper candidate sources: fetches candidates (evidence: `omnisource/distiller/pipeline.py`)
- Paper distiller pipeline → Research taxonomy: loads taxonomy (evidence: `omnisource/distiller/pipeline.py`)
- Paper distiller pipeline → Taxonomy router: routes candidates (evidence: `omnisource/distiller/pipeline.py`)
- Paper distiller pipeline → Full-text signals: enriches text (evidence: `omnisource/distiller/pipeline.py`)
- Paper distiller pipeline → Expert reviewers: reviews candidates (evidence: `omnisource/distiller/pipeline.py`)
- Paper distiller pipeline → LLM paper reviewer: requests review (evidence: `omnisource/distiller/pipeline.py`)
- Paper distiller pipeline → Candidate selection: selects papers (evidence: `omnisource/distiller/pipeline.py`)
- LLM paper reviewer → LLM providers: calls provider (evidence: `omnisource/distiller/llm_reviewer.py`)
