How chidist80/validated-inflection-model works
Project Type and Purpose
• This is a Python-based, config-driven “Inflection Engine” that implements an ETL-style pipeline to turn raw market data (e.g., SEC filings, audio transcripts) into validated market signals.
• It’s not a simple web app or library but a full-stack data-processing system with:
• An ingestion engine (polling external sources)
• A multi-stage event pipeline (transcribe → extract → compare → validate → score → deliver)
• A backing data store (PostgreSQL + Redis)
• A user-facing dashboard (Streamlit)
File Structure Analysis
Top-level directories reveal architectural boundaries:
• config/ : YAML definitions driving data sources, universe settings, weights, rate limits, and LLM prompts
• data/ : sample raw SEC filings and extracted text
• docker/ : Dockerfiles and docker-compose for Postgres, Redis, and service containers
• migrations/ : SQL scripts for schema evolution
• scripts/ : helpers to bootstrap DB, backfill data, run locally
• src/ : core Python code, subdivided into:
• app/ : Streamlit application
• ingest/ : connectors to SEC, news, audio discovery
• transcribe/ : audio preprocessing and Whisper runner
• nlp/ : document chunking, embedding, LLM-based extraction
• compare/, validate/, score/ : signal-processing and business logic
• db/ : connection, DAO, ORM models
• universe/ : stock universe builder
• notifications/ : Slack, email alerts
• jobs/ : CLI entry points orchestrating each pipeline phase
• tests/ : unit and integration tests
README Insights
• Confirms the seven-step flow: Ingest → Transcribe → Extract → Compare → Validate → Score → Deliver
• Lists external dependencies (SEC EDGAR, Ollama LLM, sentence-transformers) and backing services (Postgres, Redis)
• Describes “Event Pipeline” as the heart of the system
Main Components
a. Ingestion Engine
• Polls external sources: SEC EDGAR, news feeds, audio repositories
• Implemented in src/ingest and kicked off by src/jobs/run_sec_poll
• Writes raw filings (HTML, text) to ‘data/’ (or directly into staging tables via DAO)
b. Queue / Cache Layer
• Redis for job queueing, rate-limiting, caching intermediate results
c. Event Pipeline Workers
• A series of worker modules (jobs/run_event_pipeline) that:
1. Transcribe audio (src/transcribe)
2. Chunk and embed text (src/nlp/chunk, embed)
3. Extract signals via LLM or heuristics (src/nlp/llm_extract, filing_extract)
4. Compare against baselines and deltas (src/compare)
5. Validate and corroborate signals (src/validate)
6. Score and tier signals (src/score)
7. Persist results to PostgreSQL (src/db/dao, models) and raise alerts (src/notifications)
d. Data Store
e. External Services
• Ollama LLM runtime for local embeddings/extraction
• Sentence-Transformers models (local cache)
• SEC EDGAR API, news APIs
f. User Interface
• Streamlit app (src/app/streamlit_app.py)
• Connects to Postgres (read-only) and Redis for live metrics
Relationships and Data Flow
• Redis Queue → Event Pipeline Workers (fan-out)
• Workers → Postgres (write) and Redis (cache intermediate tokens, rate limits)
• Postgres → Streamlit UI (read)
• Workers → Notification services (Slack/email)
• Docker Compose orchestrates Postgres, Redis, pipeline containers, UI container
Architectural Patterns & Principles
• Modular pipeline (clearly separated stages)
• Config-driven logic (all behavior controlled by YAML under config/)
• Single responsibility per worker module (e.g., transcribe vs extract vs validate)
• Use of migration scripts for schema evolution (migrations/)
• Local LLM runtime for privacy and zero-API reliance
Essential Elements to Capture
• Redis used both for caching and as a job queue
• Local LLM runtime (Ollama) feeding into the extract stage
• Streamlit UI reading directly from Postgres (with optional Redis caching)
• Notification endpoints branching off scoring/alerts