# orville-wright/bespin architecture

> Architecture diagram of the GitHub repository orville-wright/bespin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/orville-wright/bespin
- Repository: https://github.com/orville-wright/bespin

## Overview

#### Purpose and Operating Model
Bespin is a Python-based personal quantitative research system, not a trade-execution application. It gathers market, fundamentals, filings, and financial-news data; applies NLP sentiment analysis; and is intended to support evidence-rich candidate opportunities and dossiers. The primary project description is in `README.md`; roadmap and data-engine requirements are further captured in `docs/PRD-SPEC_data_engines_01.txt`.

#### Entry Points and Pipeline Coordination
The repository is script-oriented, with likely top-level execution and orchestration entry points in `main.py`, `ml_yf_nlp_orchestrator.py`, and `ml_yf_nlp_news_engine.py`. These coordinate independently runnable collection, transformation, and ML stages rather than exposing a web service. Supporting operational scripts include `dump_db.py` for datastore inspection/export and `aop.py`/`md_aop.py` for analysis or processing workflows.

#### Market and Fundamental Data Acquisition
Source-specific adapters isolate external financial data APIs and feeds. Fundamental and market-data connectors reside under `data_engines_fundamentals/`, including `data_engines_fundamentals/sec_md.py`, `data_engines_fundamentals/fred_md.py`, and `data_engines_fundamentals/polygon_md.py`. The system also has standalone market collectors such as `nasdaq_quotes.py` and `marketwatch_md.py`. This boundary is where API keys, provider formats, rate limits, and source reliability are handled; requirements are documented in `docs/api_key_requirements.md`.

#### News Discovery and Web Crawling
News ingestion is split between publisher-specific adapters and Yahoo Finance-oriented pipelines. Publisher engines are under `data_engines_news/`, for example `data_engines_news/benzinga_news.py`, `data_engines_news/investing_news.py`, and `data_engines_news/forbes_news.py`. `craw4ai_news.py` provides a crawler-facing collection path, while `y_stocknews.py` and `y_newsloop.py` suggest Yahoo/news scanning workflows. Source schemas establish the normalization boundary, e.g. `json/BENZINGA_crawl4ai_schema.json` and `json/YF_sym_article_schema.json`.

#### NLP and Sentiment Processing
The ML layer turns collected article text and URLs into sentiment-oriented outputs. Core modules are `ml_sentiment.py`, `ml_cvbow.py`, and `ml_urlhinter.py`; the Yahoo-specific orchestration path is rooted at `ml_yf_nlp_news_engine.py`. A parallel or newer crawler-based implementation is preserved in `yfnews_NEW/ml_yf_news_c4.py` and `yfnews_NEW/ml_yf_nlp_reader_c4.py`. Bundled NLTK corpora/tokenizer assets in `nltk_data/` allow local text preprocessing without relying on runtime downloads.

#### Persistence, Evidence, and Graph Modeling
LMDB is the explicit local key-value persistence technology, centered on `datastore_eng_LMDB.py`, with operational access in `dump_db.py` and design notes in `README_datastore_eng_LMDB.md`. `neo4j_graphdb.py` introduces a separate graph-database boundary, likely intended to represent relationships among symbols, articles, events, and candidate evidence. The intended architecture is therefore: raw provider/crawler results → schema-normalized records → NLP-derived annotations → persisted evidence and graph relationships for later candidate reasoning.

#### Candidate-Oriented Analysis Boundary
Several symbol- and event-focused scripts form the opportunity-discovery layer: `y_topgainers.py`, `y_daylosers.py`, `y_smallcaps.py`, and `y_techevents.py`. `shallow_logic.py` and `xop.py` appear to hold higher-level selection or operational logic. Per the project model, these stages should surface and enrich candidate opportunities rather than produce autonomous buy/sell actions; thesis generation, invalidation, aging, execution tracking, and multi-user domain workflows remain roadmap concepts rather than demonstrated entry-point subsystems.

#### Runtime and External Dependencies
The material runtime is local Python, with dependencies declared in `pyproject.toml`, `requirements.txt`, and locked in `uv.lock`. External boundaries include financial APIs/providers (SEC, FRED, Polygon and other adapters), publisher/Yahoo Finance web content, Crawl4AI-style crawling, LMDB local storage, and Neo4j graph storage. GitHub Actions workflows in `.github/workflows/` provide repository automation but are not part of the analytical runtime.

Main entry points: [`datastore_eng_LMDB.py`](https://github.com/orville-wright/bespin/blob/main/datastore_eng_LMDB.py) (LMDB evidence store), [`ml_yf_nlp_news_engine.py`](https://github.com/orville-wright/bespin/blob/main/ml_yf_nlp_news_engine.py) (News NLP engine), [`json/BENZINGA_crawl4ai_schema.json`](https://github.com/orville-wright/bespin/blob/main/json/BENZINGA_crawl4ai_schema.json) (Record schemas), [`craw4ai_news.py`](https://github.com/orville-wright/bespin/blob/main/craw4ai_news.py) (Web crawler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline control"]
  node_main["Main runner<br/>Python entry point<br/>[main.py]"]
  node_yf_orchestrator["Yahoo NLP orchestrator<br/>Pipeline orchestrator"]
  node_news_engine["News NLP engine<br/>Pipeline stage"]
end

subgraph group_acquisition["Data acquisition"]
  node_fundamental_adapters["Fundamental adapters<br/>Provider connectors"]
  node_market_collectors["Market collectors<br/>Quote collectors<br/>[nasdaq_quotes.py]"]
  node_publisher_engines["Publisher news engines<br/>News adapters"]
  node_yahoo_news["Yahoo news scanning<br/>News collector<br/>[y_newsloop.py]"]
  node_crawler["Web crawler<br/>Crawler collector<br/>[craw4ai_news.py]"]
end

subgraph group_processing["Normalization &amp; NLP"]
  node_schemas["Record schemas<br/>Normalization contracts"]
  node_sentiment["Sentiment analysis<br/>NLP module<br/>[ml_sentiment.py]"]
  node_text_features["Text feature extraction<br/>NLP modules<br/>[ml_cvbow.py]"]
  node_crawler_nlp["Crawler NLP path<br/>Alternate NLP pipeline"]
end

subgraph group_analysis["Candidate research"]
  node_candidates["Candidate discovery<br/>Research screening<br/>[y_topgainers.py]"]
  node_selection_logic["Selection logic<br/>Research logic<br/>[shallow_logic.py]"]
end

subgraph group_persistence["Evidence storage"]
  node_lmdb[("LMDB evidence store<br/>Local key-value store")]
  node_neo4j[("Neo4j evidence graph<br/>Graph database adapter<br/>[neo4j_graphdb.py]")]
  node_db_dump["Datastore export<br/>Operational tool<br/>[dump_db.py]"]
end

node_providers(("Financial data providers<br/>External APIs"))
node_web_sources(("Publishers &amp; Yahoo Finance<br/>External web content"))

node_main -->|"runs"| node_yf_orchestrator
node_yf_orchestrator -->|"coordinates"| node_news_engine
node_providers -->|"API data"| node_fundamental_adapters
node_providers -->|"quotes"| node_market_collectors
node_web_sources -->|"articles"| node_publisher_engines
node_web_sources -->|"news feed"| node_yahoo_news
node_web_sources -->|"web pages"| node_crawler
node_fundamental_adapters -->|"persists records"| node_lmdb
node_market_collectors -->|"persists quotes"| node_lmdb
node_publisher_engines -->|"normalizes"| node_schemas
node_yahoo_news -->|"normalizes"| node_schemas
node_crawler -->|"normalizes"| node_schemas
node_schemas -->|"article records"| node_news_engine
node_news_engine -->|"article text"| node_sentiment
node_news_engine -->|"text and URLs"| node_text_features
node_crawler -->|"crawl results"| node_crawler_nlp
node_sentiment -->|"annotations"| node_lmdb
node_text_features -->|"derived features"| node_lmdb
node_crawler_nlp -->|"enriched articles"| node_lmdb
node_lmdb -->|"evidence relationships"| node_neo4j
node_lmdb -->|"research inputs"| node_candidates
node_candidates -->|"candidate signals"| node_selection_logic
node_selection_logic -->|"links evidence"| node_neo4j
node_lmdb -->|"inspect/export"| node_db_dump

click node_main "https://github.com/orville-wright/bespin/blob/main/main.py"
click node_yf_orchestrator "https://github.com/orville-wright/bespin/blob/main/ml_yf_nlp_orchestrator.py"
click node_news_engine "https://github.com/orville-wright/bespin/blob/main/ml_yf_nlp_news_engine.py"
click node_market_collectors "https://github.com/orville-wright/bespin/blob/main/nasdaq_quotes.py"
click node_yahoo_news "https://github.com/orville-wright/bespin/blob/main/y_newsloop.py"
click node_crawler "https://github.com/orville-wright/bespin/blob/main/craw4ai_news.py"
click node_schemas "https://github.com/orville-wright/bespin/blob/main/json/BENZINGA_crawl4ai_schema.json"
click node_sentiment "https://github.com/orville-wright/bespin/blob/main/ml_sentiment.py"
click node_text_features "https://github.com/orville-wright/bespin/blob/main/ml_cvbow.py"
click node_candidates "https://github.com/orville-wright/bespin/blob/main/y_topgainers.py"
click node_selection_logic "https://github.com/orville-wright/bespin/blob/main/shallow_logic.py"
click node_lmdb "https://github.com/orville-wright/bespin/blob/main/datastore_eng_LMDB.py"
click node_neo4j "https://github.com/orville-wright/bespin/blob/main/neo4j_graphdb.py"
click node_db_dump "https://github.com/orville-wright/bespin/blob/main/dump_db.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_yf_orchestrator,node_news_engine toneBlue
class node_fundamental_adapters,node_market_collectors,node_publisher_engines,node_yahoo_news,node_crawler toneAmber
class node_schemas,node_sentiment,node_text_features,node_crawler_nlp toneMint
class node_candidates,node_selection_logic toneRose
class node_lmdb,node_neo4j,node_db_dump toneIndigo
class node_providers,node_web_sources toneNeutral
```

## Components

### Pipeline control

- [Main runner](https://github.com/orville-wright/bespin/blob/main/main.py): Python entry point · `main.py`
- [Yahoo NLP orchestrator](https://github.com/orville-wright/bespin/blob/main/ml_yf_nlp_orchestrator.py): Pipeline orchestrator · `ml_yf_nlp_orchestrator.py`
- [News NLP engine](https://github.com/orville-wright/bespin/blob/main/ml_yf_nlp_news_engine.py): Pipeline stage · `ml_yf_nlp_news_engine.py`

### Data acquisition

- **Fundamental adapters**: Provider connectors · SEC, FRED, and Polygon source adapters.
- [Market collectors](https://github.com/orville-wright/bespin/blob/main/nasdaq_quotes.py): Quote collectors · `nasdaq_quotes.py`
- **Publisher news engines**: News adapters · Publisher-specific financial-news collection.
- [Yahoo news scanning](https://github.com/orville-wright/bespin/blob/main/y_newsloop.py): News collector · `y_newsloop.py`
- [Web crawler](https://github.com/orville-wright/bespin/blob/main/craw4ai_news.py): Crawler collector · `craw4ai_news.py`

### Normalization & NLP

- [Record schemas](https://github.com/orville-wright/bespin/blob/main/json/BENZINGA_crawl4ai_schema.json): Normalization contracts · Defines normalized crawl and article records. · `json/BENZINGA_crawl4ai_schema.json`
- [Sentiment analysis](https://github.com/orville-wright/bespin/blob/main/ml_sentiment.py): NLP module · `ml_sentiment.py`
- [Text feature extraction](https://github.com/orville-wright/bespin/blob/main/ml_cvbow.py): NLP modules · Bag-of-words and URL/text interpretation. · `ml_cvbow.py`
- **Crawler NLP path**: Alternate NLP pipeline

### Candidate research

- [Candidate discovery](https://github.com/orville-wright/bespin/blob/main/y_topgainers.py): Research screening · Surfaces opportunities; does not execute trades. · `y_topgainers.py`
- [Selection logic](https://github.com/orville-wright/bespin/blob/main/shallow_logic.py): Research logic · `shallow_logic.py`

### Evidence storage

- [LMDB evidence store](https://github.com/orville-wright/bespin/blob/main/datastore_eng_LMDB.py): Local key-value store · Persists collected and enriched evidence. · `datastore_eng_LMDB.py`
- [Neo4j evidence graph](https://github.com/orville-wright/bespin/blob/main/neo4j_graphdb.py): Graph database adapter · Models relationships among symbols, articles, events, and evidence. · `neo4j_graphdb.py`
- [Datastore export](https://github.com/orville-wright/bespin/blob/main/dump_db.py): Operational tool · `dump_db.py`

### Other components

- **Financial data providers**: External APIs · Market and fundamentals sources with provider constraints.
- **Publishers & Yahoo Finance**: External web content

## Connections

- Main runner → Yahoo NLP orchestrator: runs
- Yahoo NLP orchestrator → News NLP engine: coordinates
- Financial data providers → Fundamental adapters: API data
- Financial data providers → Market collectors: quotes
- Publishers & Yahoo Finance → Publisher news engines: articles
- Publishers & Yahoo Finance → Yahoo news scanning: news feed
- Publishers & Yahoo Finance → Web crawler: web pages
- Fundamental adapters → LMDB evidence store: persists records
- Market collectors → LMDB evidence store: persists quotes
- Publisher news engines → Record schemas: normalizes
- Yahoo news scanning → Record schemas: normalizes
- Web crawler → Record schemas: normalizes
- Record schemas → News NLP engine: article records
- News NLP engine → Sentiment analysis: article text
- News NLP engine → Text feature extraction: text and URLs
- Web crawler → Crawler NLP path: crawl results
- Sentiment analysis → LMDB evidence store: annotations
- Text feature extraction → LMDB evidence store: derived features
- Crawler NLP path → LMDB evidence store: enriched articles
- LMDB evidence store → Neo4j evidence graph: evidence relationships
- LMDB evidence store → Candidate discovery: research inputs
- Candidate discovery → Selection logic: candidate signals
- Selection logic → Neo4j evidence graph: links evidence
- LMDB evidence store → Datastore export: inspect/export