orville-wright/bespin

Purpose and Operating Model

Bespin is a Python-based personal quantitative research system, not a trade-execution application. It gathers market, fundamentals, filings, and financial-news data; applies NLP sentiment analysis; and is intended to support evidence-rich candidate opportunities and dossiers. The primary project description is in README.md; roadmap and data-engine requirements are further captured in docs/PRD-SPEC_data_engines_01.txt.

Entry Points and Pipeline Coordination

The repository is script-oriented, with likely top-level execution and orchestration entry points in main.py, ml_yf_nlp_orchestrator.py, and ml_yf_nlp_news_engine.py. These coordinate independently runnable collection, transformation, and ML stages rather than exposing a web service. Supporting operational scripts include dump_db.py for datastore inspection/export and aop.py/md_aop.py for analysis or processing workflows.

Market and Fundamental Data Acquisition

Source-specific adapters isolate external financial data APIs and feeds. Fundamental and market-data connectors reside under data_engines_fundamentals/, including data_engines_fundamentals/sec_md.py, data_engines_fundamentals/fred_md.py, and data_engines_fundamentals/polygon_md.py. The system also has standalone market collectors such as nasdaq_quotes.py and marketwatch_md.py. This boundary is where API keys, provider formats, rate limits, and source reliability are handled; requirements are documented in docs/api_key_requirements.md.

News Discovery and Web Crawling

News ingestion is split between publisher-specific adapters and Yahoo Finance-oriented pipelines. Publisher engines are under data_engines_news/, for example data_engines_news/benzinga_news.py, data_engines_news/investing_news.py, and data_engines_news/forbes_news.py. craw4ai_news.py provides a crawler-facing collection path, while y_stocknews.py and y_newsloop.py suggest Yahoo/news scanning workflows. Source schemas establish the normalization boundary, e.g. json/BENZINGA_crawl4ai_schema.json and json/YF_sym_article_schema.json.

NLP and Sentiment Processing

The ML layer turns collected article text and URLs into sentiment-oriented outputs. Core modules are ml_sentiment.py, ml_cvbow.py, and ml_urlhinter.py; the Yahoo-specific orchestration path is rooted at ml_yf_nlp_news_engine.py. A parallel or newer crawler-based implementation is preserved in yfnews_NEW/ml_yf_news_c4.py and yfnews_NEW/ml_yf_nlp_reader_c4.py. Bundled NLTK corpora/tokenizer assets in nltk_data/ allow local text preprocessing without relying on runtime downloads.

Persistence, Evidence, and Graph Modeling

LMDB is the explicit local key-value persistence technology, centered on datastore_eng_LMDB.py, with operational access in dump_db.py and design notes in README_datastore_eng_LMDB.md. neo4j_graphdb.py introduces a separate graph-database boundary, likely intended to represent relationships among symbols, articles, events, and candidate evidence. The intended architecture is therefore: raw provider/crawler results → schema-normalized records → NLP-derived annotations → persisted evidence and graph relationships for later candidate reasoning.

Candidate-Oriented Analysis Boundary

Several symbol- and event-focused scripts form the opportunity-discovery layer: y_topgainers.py, y_daylosers.py, y_smallcaps.py, and y_techevents.py. shallow_logic.py and xop.py appear to hold higher-level selection or operational logic. Per the project model, these stages should surface and enrich candidate opportunities rather than produce autonomous buy/sell actions; thesis generation, invalidation, aging, execution tracking, and multi-user domain workflows remain roadmap concepts rather than demonstrated entry-point subsystems.

Runtime and External Dependencies

The material runtime is local Python, with dependencies declared in pyproject.toml, requirements.txt, and locked in uv.lock. External boundaries include financial APIs/providers (SEC, FRED, Polygon and other adapters), publisher/Yahoo Finance web content, Crawl4AI-style crawling, LMDB local storage, and Neo4j graph storage. GitHub Actions workflows in .github/workflows/ provide repository automation but are not part of the analytical runtime.