# ronnyling/income_research_os architecture

> Architecture diagram of the GitHub repository ronnyling/income_research_os, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ronnyling/income_research_os
- Repository: https://github.com/ronnyling/income_research_os

## Overview

1. **What this repo is**
   - `incomos` is an investment research pipeline, not an execution/trading system.
   - Its main output is a ranked, human-reviewed funnel of income stocks: universe → screen → KIV watchlist → candidates → finalists → decision queue.

2. **Primary orchestration path**
   - The end-to-end run is driven by `scripts/run_funnel.py`.
   - That script pulls together the modules under `src/incomos/funnel`, `screening`, `scoring`, `llm`, `macro`, `data`, and `persistence`.
   - The architecture is stage-based and mostly batch-oriented, with expensive work deferred until later stages.

3. **Universe and data ingestion boundary**
   - `src/incomos/data` is the external-data layer.
   - It contains fetchers for EDGAR/SEC filings, market prices, FRED macro series, Bursa Malaysia data, NOBL holdings, and ticker universe construction.
   - This is the main boundary between the system and external services such as EDGAR, yfinance, FRED, and Bursa sources.

4. **Stage 0–1: fast deterministic screening**
   - `src/incomos/screening/stage01.py` performs the first hard filter over the stock universe.
   - It uses structured financial facts to reject names that fail basic quality gates such as dividend history, FCF, payout logic, market cap, and ROIC-style checks.
   - This stage is intentionally cheap and rule-based; it reduces the universe before any LLM work.

5. **Stage 1–2: entry attractiveness ranking**
   - `src/incomos/funnel/entry_attractiveness.py` ranks screened stocks on a 0–100 attractiveness score.
   - The score combines drawdown, yield expansion, dividend growth, valuation/FCF quality, and trend.
   - This stage decides which stocks enter the KIV basket, so it is the bridge between screening and ongoing monitoring.

6. **KIV basket as the active watch layer**
   - `src/incomos/funnel/kiv.py` and `refresh.py` manage the watchlist lifecycle.
   - KIV is not passive storage: names can be promoted, demoted to dormant, or rejected based on TTL expiry, new bad filings, dividend risk, or deteriorating fundamentals.
   - This layer is where daily/weekly refresh logic concentrates, so the rest of the pipeline does not recompute everything every run.

7. **Stage 2–4: LLM-assisted analysis**
   - `src/incomos/llm/mimo.py` and `schemas.py` encapsulate MiMo 2.5 use.
   - The code only invokes the model after cheaper rule-based tools have run, and its outputs are schema-validated before they affect scoring.
   - In the architecture diagram, this should be shown as an expensive, gated branch that consumes filing text and structured context, then returns classification and narrative reasoning.

8. **Scoring engine and decision model**
   - `src/incomos/scoring` contains the business, income, dip, oversold, and aggregate scoring logic.
   - `engine.py` combines these sub-scores into the final opportunity score, while `dip.py`, `business.py`, `income.py`, and `oversold.py` handle component calculations.
   - The scoring model is config-driven via `src/incomos/core/config.py`, so thresholds and weights are not hardcoded.

9. **Macro context as a cross-cutting input**
   - `src/incomos/macro/regime.py` supplies a regime view used in later-stage interpretation.
   - Macro inputs come from FRED-style series and are treated as a decision context layer, not as a primary driver of the whole universe screen.
   - In the diagram, this is a side input into candidate/finalist evaluation, not a standalone pipeline branch.

10. **Persistence and checkpointing**
   - `src/incomos/persistence` holds the PostgreSQL boundary: session management, DB setup, and query/upsert logic.
   - The funnel also writes a checkpoint and human-readable MiMo results so later runs can regenerate reports without redoing all network and LLM work.
   - This is a useful artifact boundary: raw data and intermediate state are persisted separately from final report output.

11. **Backtest and validation layer**
   - `src/incomos/backtest/validation.py` and `mcp_servers/backtest_mcp.py` support forward-return validation and calibration reporting.
   - This layer evaluates whether score buckets and classifications have predictive value.
   - It sits alongside the main research pipeline rather than inside the live funnel path.

12. **MCP server decomposition**
   - The `mcp_servers/` package exposes domain tools as separate servers: market data, SEC filings, macro regime, research store, backtest, and Bursa Malaysia.
   - Architecturally, these are tool-facing wrappers around the internal modules, giving an agent or client a narrow domain API instead of direct access to internal code.
   - If you are drawing the system, treat them as service façades over the same core domain logic.

13. **Execution model and concurrency**
   - The repo is Python-based and uses `ThreadPoolExecutor` heavily for parallel fetching and batch MiMo work.
   - Parallelism matters architecturally because fetching prices, downloading filings, and classifying batches are all concurrency-aware.
   - This reduces wall-clock time, but the pipeline still preserves stage ordering and validation gates.

14. **User-facing surfaces**
   - `src/incomos/api/main.py` provides a FastAPI surface, while `docs/index.html` is the generated report output.
   - The primary human interaction is still CLI-driven and report-based, not a transactional web workflow.
   - So the cleanest diagram is: CLI/API entrypoints → funnel pipeline → optional DB/report artifacts.

15. **Key architectural constraints**
   - `src/incomos/core` centralizes typed config, exceptions, and shared types, which enforces the repo’s “no silent fallback” and “no hardcoded threshold” design.
   - Those constraints are important to show because they define how data failure propagates: missing inputs become typed errors, not default scores.
   - That makes the pipeline deterministic, auditable, and suitable for human-in-the-loop research rather than automated trading.

Main entry points: [`scripts/run_funnel.py`](https://github.com/ronnyling/income_research_os/blob/main/scripts/run_funnel.py) (Run funnel), [`src/incomos/scoring/engine.py`](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/scoring/engine.py) (Score engine), [`mcp_servers/__init__.py`](https://github.com/ronnyling/income_research_os/blob/main/mcp_servers/__init__.py) (MCP tools), [`src/incomos/llm/mimo.py`](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/llm/mimo.py) (MiMo analysis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_entry["Entrypoints"]
  node_node_run_funnel["Run funnel<br/>CLI orchestrator<br/>[run_funnel.py]"]
  node_node_api_main["API<br/>[main.py]"]
  node_node_mcp_servers["MCP tools<br/>tool façades<br/>[__init__.py]"]
end

subgraph group_grp_pipeline["Research Funnel"]
  node_node_stage01["Stage 01<br/>rule screen<br/>[stage01.py]"]
  node_node_entry_attr["Attractiveness<br/>ranker"]
  node_node_kiv["KIV basket<br/>watchlist<br/>[kiv.py]"]
  node_node_refresh["Refresh<br/>[refresh.py]"]
  node_node_dip_trigger["Dip trigger<br/>entry signal<br/>[dip_trigger.py]"]
end

subgraph group_grp_data["Data Sources"]
  node_node_universe[("Universe<br/>ticker set<br/>[universe.py]")]
  node_node_market_data[("Market data<br/>price fetchers<br/>[market.py]")]
  node_node_filings[("Filings<br/>SEC/EDGAR ingest<br/>[filings.py]")]
  node_node_bursa[("Bursa data<br/>regional ingest<br/>[bursa.py]")]
  node_node_fred[("Macro series<br/>FRED ingest<br/>[fred.py]")]
  node_node_nobl[("NOBL holdings<br/>index ingest<br/>[nobl.py]")]
end

subgraph group_grp_analysis["Analysis"]
  node_node_mimo{{"MiMo analysis<br/>LLM gate<br/>[mimo.py]"}}
  node_node_schemas["Schemas<br/>output schema<br/>[schemas.py]"]
  node_node_scoring_engine["Score engine<br/>aggregate scorer<br/>[engine.py]"]
  node_node_scoring_parts["Score parts<br/>component scorers<br/>[business.py]"]
  node_node_macro_regime(("Regime<br/>macro context<br/>[regime.py]"))
  node_node_validation["Validation<br/>[validation.py]"]
  node_node_config["Config<br/>typed settings<br/>[config.py]"]
  node_node_core["Core types<br/>shared contracts<br/>[types.py]"]
end

subgraph group_grp_store["State & Artifacts"]
  node_node_db[("Postgres<br/>persistence layer<br/>[db.py]")]
  node_node_queries["Queries<br/>upserts & reads<br/>[queries.py]"]
  node_node_session["Session<br/>[session.py]"]
  node_node_docs["Report<br/>HTML output<br/>[index.html]"]
end

node_node_run_funnel -->|"build universe"| node_node_universe
node_node_run_funnel -->|"screen"| node_node_stage01
node_node_stage01 -->|"pass names"| node_node_entry_attr
node_node_entry_attr -->|"enter watchlist"| node_node_kiv
node_node_kiv -->|"refresh"| node_node_refresh
node_node_refresh -->|"evaluate dips"| node_node_dip_trigger
node_node_run_funnel -->|"fetch filings"| node_node_filings
node_node_run_funnel -->|"fetch prices"| node_node_market_data
node_node_run_funnel -->|"fetch regional data"| node_node_bursa
node_node_run_funnel -->|"load benchmark"| node_node_nobl
node_node_run_funnel -->|"load macro"| node_node_fred
node_node_run_funnel -->|"LLM analysis"| node_node_mimo
node_node_mimo -->|"validate output"| node_node_schemas
node_node_mimo -->|"assist scoring"| node_node_scoring_engine
node_node_scoring_parts -->|"component scores"| node_node_scoring_engine
node_node_config -->|"weights & thresholds"| node_node_scoring_engine
node_node_config -->|"screen rules"| node_node_stage01
node_node_macro_regime -->|"context input"| node_node_scoring_engine
node_node_db -->|"backing store"| node_node_queries
node_node_session -->|"manage tx"| node_node_db
node_node_run_funnel -->|"checkpoint state"| node_node_db
node_node_run_funnel -->|"write report"| node_node_docs
node_node_validation -.->|"calibrate"| node_node_scoring_engine
node_node_api_main -.->|"invoke pipeline"| node_node_run_funnel
node_node_mcp_servers -.->|"filing tools"| node_node_filings
node_node_mcp_servers -.->|"market tools"| node_node_market_data
node_node_mcp_servers -.->|"macro tools"| node_node_macro_regime
node_node_mcp_servers -.->|"backtest tools"| node_node_validation

click node_node_run_funnel "https://github.com/ronnyling/income_research_os/blob/main/scripts/run_funnel.py"
click node_node_api_main "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/api/main.py"
click node_node_mcp_servers "https://github.com/ronnyling/income_research_os/blob/main/mcp_servers/__init__.py"
click node_node_universe "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/universe.py"
click node_node_market_data "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/market.py"
click node_node_filings "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/filings.py"
click node_node_bursa "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/bursa.py"
click node_node_fred "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/fred.py"
click node_node_nobl "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/nobl.py"
click node_node_stage01 "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/screening/stage01.py"
click node_node_entry_attr "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/funnel/entry_attractiveness.py"
click node_node_kiv "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/funnel/kiv.py"
click node_node_refresh "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/funnel/refresh.py"
click node_node_dip_trigger "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/funnel/dip_trigger.py"
click node_node_mimo "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/llm/mimo.py"
click node_node_schemas "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/llm/schemas.py"
click node_node_scoring_engine "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/scoring/engine.py"
click node_node_scoring_parts "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/scoring/business.py"
click node_node_macro_regime "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/macro/regime.py"
click node_node_validation "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/backtest/validation.py"
click node_node_config "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/core/config.py"
click node_node_core "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/core/types.py"
click node_node_db "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/persistence/db.py"
click node_node_queries "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/persistence/queries.py"
click node_node_session "https://github.com/ronnyling/income_research_os/blob/main/src/incomos/persistence/session.py"
click node_node_docs "https://github.com/ronnyling/income_research_os/blob/main/docs/index.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_run_funnel,node_node_api_main,node_node_mcp_servers toneBlue
class node_node_stage01,node_node_entry_attr,node_node_kiv,node_node_refresh,node_node_dip_trigger toneAmber
class node_node_universe,node_node_market_data,node_node_filings,node_node_bursa,node_node_fred,node_node_nobl toneMint
class node_node_mimo,node_node_schemas,node_node_scoring_engine,node_node_scoring_parts,node_node_macro_regime,node_node_validation,node_node_config,node_node_core toneRose
class node_node_db,node_node_queries,node_node_session,node_node_docs toneIndigo
```

## Components

### Entrypoints

CLI, API, and tool-facing surfaces that start or expose the research system.

- [Run funnel](https://github.com/ronnyling/income_research_os/blob/main/scripts/run_funnel.py): CLI orchestrator · End-to-end batch driver for the income research funnel. · `scripts/run_funnel.py`
- [API](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/api/main.py): FastAPI surface · HTTP entrypoint for programmatic access to the research system. · `src/incomos/api/main.py`
- [MCP tools](https://github.com/ronnyling/income_research_os/blob/main/mcp_servers/__init__.py): tool façades · Domain servers exposing narrow APIs over core research modules. · `mcp_servers/__init__.py`

### Research Funnel

Stage-based pipeline from universe screening through watchlist, scoring, and decision outputs.

- [Stage 01](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/screening/stage01.py): rule screen · Fast deterministic filter for dividend quality, cash flow, payout, size, and return metrics. · `src/incomos/screening/stage01.py`
- [Attractiveness](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/funnel/entry_attractiveness.py): ranker · Scores screened names for entry into the KIV basket. · `src/incomos/funnel/entry_attractiveness.py`
- [KIV basket](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/funnel/kiv.py): watchlist · Active watch layer with promotion, dormancy, and rejection states. · `src/incomos/funnel/kiv.py`
- [Refresh](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/funnel/refresh.py): watchlist refresh · Daily or weekly KIV lifecycle updates and re-evaluation logic. · `src/incomos/funnel/refresh.py`
- [Dip trigger](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/funnel/dip_trigger.py): entry signal · Detects dip-based entry conditions for watchlist names. · `src/incomos/funnel/dip_trigger.py`

### Data Sources

External-data boundary and universe construction inputs.

- [Universe](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/universe.py): ticker set · Constructs the investable stock universe from source data. · `src/incomos/data/universe.py`
- [Market data](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/market.py): price fetchers · Market prices and related quote data used by screening and scoring. · `src/incomos/data/market.py`
- [Filings](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/filings.py): SEC/EDGAR ingest · Fetches and normalizes SEC/EDGAR filings and filing text. · `src/incomos/data/filings.py`
- [Bursa data](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/bursa.py): regional ingest · Bursa Malaysia-specific market and company data boundary. · `src/incomos/data/bursa.py`
- [Macro series](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/fred.py): FRED ingest · Macro time series used as contextual inputs. · `src/incomos/data/fred.py`
- [NOBL holdings](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/data/nobl.py): index ingest · Dividend aristocrat-style benchmark holdings input. · `src/incomos/data/nobl.py`

### Analysis

LLM, scoring, macro, and validation logic that interpret and rank candidates.

- [MiMo analysis](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/llm/mimo.py): LLM gate · Schema-validated LLM analysis over filing text and structured context. · `src/incomos/llm/mimo.py`
- [Schemas](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/llm/schemas.py): output schema · Validated response shapes for model outputs and reasoning. · `src/incomos/llm/schemas.py`
- [Score engine](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/scoring/engine.py): aggregate scorer · Combines component scores into the final opportunity score. · `src/incomos/scoring/engine.py`
- [Score parts](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/scoring/business.py): component scorers · Business, income, dip, and oversold score calculations. · `src/incomos/scoring/business.py`
- [Regime](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/macro/regime.py): macro context · Macro regime view used in later-stage interpretation. · `src/incomos/macro/regime.py`
- [Validation](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/backtest/validation.py): backtest/validation · Forward-return validation and calibration reporting for score buckets. · `src/incomos/backtest/validation.py`
- [Config](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/core/config.py): typed settings · Central configuration for thresholds, weights, and runtime behavior. · `src/incomos/core/config.py`
- [Core types](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/core/types.py): shared contracts · Typed exceptions and shared domain types that enforce explicit failures. · `src/incomos/core/types.py`

### State & Artifacts

Persistence, checkpoints, and report artifacts.

- [Postgres](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/persistence/db.py): persistence layer · Database boundary for sessions, setup, and durable research state. · `src/incomos/persistence/db.py`
- [Queries](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/persistence/queries.py): upserts & reads · Query and upsert logic for research records and funnel state. · `src/incomos/persistence/queries.py`
- [Session](https://github.com/ronnyling/income_research_os/blob/main/src/incomos/persistence/session.py): db session · Session management and transaction boundary for persistence. · `src/incomos/persistence/session.py`
- [Report](https://github.com/ronnyling/income_research_os/blob/main/docs/index.html): HTML output · Human-readable generated research report artifact. · `docs/index.html`

## Connections

- Run funnel → Universe: build universe · Loads ticker universe and source inputs before screening.
- Run funnel → Stage 01: screen · Runs the first hard filter over the investable universe.
- Stage 01 → Attractiveness: pass names · Qualified names proceed to attractiveness ranking.
- Attractiveness → KIV basket: enter watchlist · Top-ranked candidates are placed into the KIV basket.
- KIV basket → Refresh: refresh · Refresh logic updates watch states and lifecycle decisions.
- Refresh → Dip trigger: evaluate dips · Watchlist refresh can trigger dip-based entry checks.
- Run funnel → Filings: fetch filings · Retrieves filing text and structured filing data.
- Run funnel → Market data: fetch prices · Pulls market data needed across screening and scoring.
- Run funnel → Bursa data: fetch regional data · Pulls Bursa-specific inputs when relevant.
- Run funnel → NOBL holdings: load benchmark · Uses NOBL holdings as a dividend reference input.
- Run funnel → Macro series: load macro · Ingests macro series for later-stage interpretation.
- Run funnel → MiMo analysis: LLM analysis · Gated expensive branch for schema-validated model reasoning.
- MiMo analysis → Schemas: validate output · Model output is checked against typed schemas before use.
- MiMo analysis → Score engine: assist scoring · LLM results can inform downstream scoring and classification.
- Score parts → Score engine: component scores · Business, income, dip, and oversold components feed the aggregate score.
- Config → Score engine: weights & thresholds · Scoring is controlled by typed configuration rather than hardcoded constants.
- Config → Stage 01: screen rules · Deterministic filters depend on configurable thresholds.
- Regime → Score engine: context input · Macro regime influences later-stage interpretation and ranking.
- Postgres → Queries: backing store · Queries operate over the PostgreSQL persistence boundary.
- Session → Postgres: manage tx · Session layer wraps database access and transactions.
- Run funnel → Postgres: checkpoint state · The funnel persists intermediate state and artifacts for reruns.
- Run funnel → Report: write report · Produces human-readable report output from the research run.
- Validation → Score engine: calibrate · Backtest validation checks score buckets and classification value.
- API → Run funnel: invoke pipeline · API surface can trigger or wrap the same funnel workflow.
- MCP tools → Filings: filing tools · Tool server exposes filing-related domain operations.
- MCP tools → Market data: market tools · Tool server exposes market data operations.
- MCP tools → Regime: macro tools · Tool server exposes macro regime operations.
- MCP tools → Validation: backtest tools · Tool server exposes validation and calibration operations.