How lang-jiaqi/omnisource works
OmniSource turns configurable research, startup, and developer-tool signals into curated, ranked reports, published locally or to GitHub Issues. The primary workflow is the track-driven collector-to-publisher pipeline; a separate CS paper distiller enriches and reviews research candidates. The graph preserves both workflows and feedback-based personalization. Some relationships are supported only by the README or SOURCE INDEX references; unsampled subsystem details and their internal call paths are not assumed.
Main entry points: main.py Pipeline orchestrator · pipeline.py Paper distiller pipeline · publishers Report publishers · curator.py Curation and shortlist
Components 30
Report workflow
- CLI
omnisource/cli.py - Track configuration
omnisource/config.py - Pipeline orchestrator
omnisource/main.py - Signal sources
omnisource/sources - Unified signals
omnisource/models.py - Signal enrichment
omnisource/enrich.py - Deduplication
omnisource/dedup.py - Report editor
omnisource/agents/editor.py - Report publishers
omnisource/publishers
Curation and ranking
- Curation and shortlist
omnisource/agents/curator.py - Quality filtering
omnisource/agents/quality.py - LLM analysis
omnisource/agents/analyst.py - Paper review pass
omnisource/agents/review_distill.py - Ranking scores
omnisource/ranking/scoring.py - Feedback personalization
omnisource/personalization.py
Paper distiller
- Paper distiller pipeline
omnisource/distiller/pipeline.py - Paper candidate sources
omnisource/distiller/sources.py - Research taxonomy
omnisource/distiller/taxonomy.py - Taxonomy router
omnisource/distiller/router.py - Full-text signals
omnisource/distiller/fulltext.py - Expert reviewers
omnisource/distiller/reviewer.py - LLM paper reviewer
omnisource/distiller/llm_reviewer.py - Candidate selection
omnisource/distiller/selection.py
State and integrations
- Repository feedback import
omnisource/repository_feedback.py - Signal memory
omnisource/memory/store.py - Fetch cache
omnisource/memory/cache.py - LLM providers
omnisource/llm/base.py
Other components
- Track operator
- GitHub Issues
- Markdown and JSONL reports
Connections 31
- Track operator → CLI: runs
README.md - CLI → Pipeline orchestrator: starts
- Pipeline orchestrator → Track configuration: loads tracks
omnisource/main.py - Pipeline orchestrator → Signal sources: collects signals
omnisource/main.py - Signal sources → Unified signals: normalizes
omnisource/models.py - Pipeline orchestrator → Signal enrichment: enriches
omnisource/main.py - Pipeline orchestrator → Deduplication: deduplicates
- Pipeline orchestrator → Curation and shortlist: shortlists
omnisource/main.py - Pipeline orchestrator → Quality filtering: filters
omnisource/main.py - Pipeline orchestrator → Paper review pass: reviews papers
omnisource/main.py - Pipeline orchestrator → LLM analysis: analyzes
- Curation and shortlist → LLM analysis: consults
omnisource/agents/curator.py - Curation and shortlist → Ranking scores: scores
omnisource/agents/curator.py - Pipeline orchestrator → Feedback personalization: personalizes
omnisource/main.py - Pipeline orchestrator → Report editor: edits report
omnisource/main.py - Pipeline orchestrator → Report publishers: publishes
omnisource/main.py - Report publishers → GitHub Issues: creates issues
README.md - Report publishers → Markdown and JSONL reports: writes reports
README.md - Pipeline orchestrator → Signal memory: uses memory
omnisource/main.py - Signal enrichment → Fetch cache: uses cache
omnisource/enrich.py - Repository feedback import → Feedback personalization: supplies events
omnisource/personalization.py - GitHub Issues → Repository feedback import: provides issue feedback
- Track operator → Paper distiller pipeline: runs distiller
- Paper distiller pipeline → Paper candidate sources: fetches candidates
omnisource/distiller/pipeline.py - Paper distiller pipeline → Research taxonomy: loads taxonomy
omnisource/distiller/pipeline.py - Paper distiller pipeline → Taxonomy router: routes candidates
omnisource/distiller/pipeline.py - Paper distiller pipeline → Full-text signals: enriches text
omnisource/distiller/pipeline.py - Paper distiller pipeline → Expert reviewers: reviews candidates
omnisource/distiller/pipeline.py - Paper distiller pipeline → LLM paper reviewer: requests review
omnisource/distiller/pipeline.py - Paper distiller pipeline → Candidate selection: selects papers
omnisource/distiller/pipeline.py - LLM paper reviewer → LLM providers: calls provider
omnisource/distiller/llm_reviewer.py