# chidist80/validated-inflection-model architecture

> Architecture diagram of the GitHub repository chidist80/validated-inflection-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chidist80/validated-inflection-model
- Repository: https://github.com/chidist80/validated-inflection-model

## Overview

##### Project Type and Purpose
- This is a Python-based, config-driven “Inflection Engine” that implements an ETL-style pipeline to turn raw market data (e.g., SEC filings, audio transcripts) into validated market signals.
- It’s not a simple web app or library but a full-stack data-processing system with:
- An ingestion engine (polling external sources)
- A multi-stage event pipeline (transcribe → extract → compare → validate → score → deliver)
- A backing data store (PostgreSQL + Redis)
- A user-facing dashboard (Streamlit)

##### File Structure Analysis
Top-level directories reveal architectural boundaries:
- config/ : YAML definitions driving data sources, universe settings, weights, rate limits, and LLM prompts
- data/ : sample raw SEC filings and extracted text
- docker/ : Dockerfiles and docker-compose for Postgres, Redis, and service containers
- migrations/ : SQL scripts for schema evolution
- scripts/ : helpers to bootstrap DB, backfill data, run locally
- src/ : core Python code, subdivided into:
- app/ : Streamlit application
- ingest/ : connectors to SEC, news, audio discovery
- transcribe/ : audio preprocessing and Whisper runner
- nlp/ : document chunking, embedding, LLM-based extraction
- compare/, validate/, score/ : signal-processing and business logic
- db/ : connection, DAO, ORM models
- universe/ : stock universe builder
- notifications/ : Slack, email alerts
- jobs/ : CLI entry points orchestrating each pipeline phase
- tests/ : unit and integration tests

##### README Insights
- Confirms the seven-step flow: Ingest → Transcribe → Extract → Compare → Validate → Score → Deliver
- Lists external dependencies (SEC EDGAR, Ollama LLM, sentence-transformers) and backing services (Postgres, Redis)
- Describes “Event Pipeline” as the heart of the system

##### Main Components
a. Ingestion Engine
- Polls external sources: SEC EDGAR, news feeds, audio repositories
- Implemented in src/ingest and kicked off by src/jobs/run_sec_poll
- Writes raw filings (HTML, text) to ‘data/’ (or directly into staging tables via DAO)
b. Queue / Cache Layer
- Redis for job queueing, rate-limiting, caching intermediate results
c. Event Pipeline Workers
- A series of worker modules (jobs/run_event_pipeline) that:
1. Transcribe audio (src/transcribe)
2. Chunk and embed text (src/nlp/chunk, embed)
3. Extract signals via LLM or heuristics (src/nlp/llm_extract, filing_extract)
4. Compare against baselines and deltas (src/compare)
5. Validate and corroborate signals (src/validate)
6. Score and tier signals (src/score)
7. Persist results to PostgreSQL (src/db/dao, models) and raise alerts (src/notifications)
d. Data Store
e. External Services
- Ollama LLM runtime for local embeddings/extraction
- Sentence-Transformers models (local cache)
- SEC EDGAR API, news APIs
f. User Interface
- Streamlit app (src/app/streamlit_app.py)
- Connects to Postgres (read-only) and Redis for live metrics

##### Relationships and Data Flow
- Redis Queue → Event Pipeline Workers (fan-out)
- Workers → Postgres (write) and Redis (cache intermediate tokens, rate limits)
- Postgres → Streamlit UI (read)
- Workers → Notification services (Slack/email)
- Docker Compose orchestrates Postgres, Redis, pipeline containers, UI container

##### Architectural Patterns & Principles
- Modular pipeline (clearly separated stages)
- Config-driven logic (all behavior controlled by YAML under config/)
- Single responsibility per worker module (e.g., transcribe vs extract vs validate)
- Use of migration scripts for schema evolution (migrations/)
- Local LLM runtime for privacy and zero-API reliance

##### Essential Elements to Capture
- Redis used both for caching and as a job queue
- Local LLM runtime (Ollama) feeding into the extract stage
- Streamlit UI reading directly from Postgres (with optional Redis caching)
- Notification endpoints branching off scoring/alerts

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems" 
        direction TB
        SEC_EDGAR[/"SEC EDGAR API"/]:::external
        NEWS_FEEDS[/"News APIs"/]:::external
        AUDIO_REPOS[/"Audio Repositories"/]:::external
    end

    %% Configuration Control Plane
    subgraph "Config Control Plane" 
        direction TB
        CONFIG[("config/ (YAML & prompts)")]:::config
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer" 
        direction TB
        INGEST_ENGINE["Ingest Engine (src/ingest/)"]:::ingestion
        RUN_SEC_POLL["run_sec_poll.py"]:::ingestion
    end

    %% Queue / Cache Layer
    subgraph "Queue / Cache Layer" 
        direction TB
        REDIS[(Redis)]:::datastore
    end

    %% Event Pipeline
    subgraph "Event Pipeline Workers" 
        direction TB
        RUN_PIPELINE["run_event_pipeline.py"]:::pipeline
        TRANSCRIBE["Transcription\n(src/transcribe/)"]:::pipeline
        CHUNK_EMBED["Chunk & Embed\n(src/nlp/)"]:::pipeline
        EXTRACT["Signal Extraction\n(src/nlp/)"]:::pipeline
        COMPARE["Comparison\n(src/compare/)"]:::pipeline
        VALIDATE["Validation\n(src/validate/)"]:::pipeline
        SCORE["Scoring\n(src/score/)"]:::pipeline
    end

    %% Data Stores
    subgraph "Data Stores" 
        direction TB
        POSTGRES[(PostgreSQL)]:::datastore
    end

    %% Notifications
    subgraph "Notifications" 
        direction TB
        ALERT_JOB["run_alerts.py"]:::pipeline
        SLACK["Slack\n(src/notifications/)"]:::pipeline
        EMAIL["Email\n(src/notifications/)"]:::pipeline
    end

    %% User Interface
    subgraph "User Interface" 
        direction TB
        STREAMLIT["Streamlit App\n(src/app/streamlit_app.py)"]:::ui
    end

    %% Deployment / Initialization
    subgraph "Deployment / Init" 
        direction TB
        DOCKER_COMPOSE["docker-compose.yml"]:::deployment
        DOCKER_APP["Dockerfile.app"]:::deployment
        DOCKER_WORKER["Dockerfile.worker"]:::deployment
        MIGRATIONS["migrations/"]:::deployment
        SCRIPTS["scripts/"]:::deployment
    end

    %% Connections
    SEC_EDGAR -->|poll| INGEST_ENGINE
    NEWS_FEEDS -->|poll| INGEST_ENGINE
    AUDIO_REPOS -->|discover| INGEST_ENGINE
    CONFIG --> INGEST_ENGINE

    INGEST_ENGINE -->|enqueue| REDIS
    RUN_SEC_POLL --> INGEST_ENGINE

    REDIS -->|trigger| RUN_PIPELINE
    CONFIG --> RUN_PIPELINE

    RUN_PIPELINE --> TRANSCRIBE
    TRANSCRIBE --> CHUNK_EMBED
    CHUNK_EMBED --> EXTRACT
    EXTRACT --> COMPARE
    COMPARE --> VALIDATE
    VALIDATE --> SCORE
    SCORE --> POSTGRES
    SCORE --> ALERT_JOB
    SCORE --> REDIS

    ALERT_JOB --> SLACK
    ALERT_JOB --> EMAIL

    POSTGRES --> STREAMLIT
    REDIS --> STREAMLIT
    CONFIG --> STREAMLIT

    DOCKER_COMPOSE --> POSTGRES
    DOCKER_COMPOSE --> REDIS
    DOCKER_COMPOSE --> RUN_PIPELINE
    DOCKER_COMPOSE --> STREAMLIT

    SCRIPTS --> MIGRATIONS
    MIGRATIONS --> POSTGRES

    %% Click Events
    click INGEST_ENGINE "https://github.com/chidist80/validated-inflection-model/tree/main/src/ingest/"
    click RUN_SEC_POLL "https://github.com/chidist80/validated-inflection-model/blob/main/src/jobs/run_sec_poll.py"
    click REDIS "https://github.com/chidist80/validated-inflection-model/blob/main/docker/docker-compose.yml"
    click RUN_PIPELINE "https://github.com/chidist80/validated-inflection-model/blob/main/src/jobs/run_event_pipeline.py"
    click TRANSCRIBE "https://github.com/chidist80/validated-inflection-model/tree/main/src/transcribe/"
    click CHUNK_EMBED "https://github.com/chidist80/validated-inflection-model/blob/main/src/nlp/chunk.py"
    click CHUNK_EMBED "https://github.com/chidist80/validated-inflection-model/blob/main/src/nlp/embed.py"
    click EXTRACT "https://github.com/chidist80/validated-inflection-model/blob/main/src/nlp/filing_extract.py"
    click EXTRACT "https://github.com/chidist80/validated-inflection-model/blob/main/src/nlp/llm_extract.py"
    click EXTRACT "https://github.com/chidist80/validated-inflection-model/blob/main/src/nlp/json_repair.py"
    click COMPARE "https://github.com/chidist80/validated-inflection-model/tree/main/src/compare/"
    click VALIDATE "https://github.com/chidist80/validated-inflection-model/tree/main/src/validate/"
    click SCORE "https://github.com/chidist80/validated-inflection-model/tree/main/src/score/"
    click ALERT_JOB "https://github.com/chidist80/validated-inflection-model/blob/main/src/jobs/run_alerts.py"
    click SLACK "https://github.com/chidist80/validated-inflection-model/blob/main/src/notifications/slack.py"
    click EMAIL "https://github.com/chidist80/validated-inflection-model/blob/main/src/notifications/emailer.py"
    click POSTGRES "https://github.com/chidist80/validated-inflection-model/blob/main/src/db/connection.py"
    click POSTGRES "https://github.com/chidist80/validated-inflection-model/blob/main/src/db/dao.py"
    click POSTGRES "https://github.com/chidist80/validated-inflection-model/blob/main/src/db/models.py"
    click STREAMLIT "https://github.com/chidist80/validated-inflection-model/blob/main/src/app/streamlit_app.py"
    click CONFIG "https://github.com/chidist80/validated-inflection-model/tree/main/config/"
    click DOCKER_COMPOSE "https://github.com/chidist80/validated-inflection-model/blob/main/docker/docker-compose.yml"
    click DOCKER_APP "https://github.com/chidist80/validated-inflection-model/blob/main/docker/Dockerfile.app"
    click DOCKER_WORKER "https://github.com/chidist80/validated-inflection-model/blob/main/docker/Dockerfile.worker"
    click MIGRATIONS "https://github.com/chidist80/validated-inflection-model/tree/main/migrations/"
    click SCRIPTS "https://github.com/chidist80/validated-inflection-model/tree/main/scripts/"

    %% Styles
    classDef external fill:#dddddd,stroke:#888,stroke-width:1px
    classDef config fill:#f9f0c1,stroke:#b59400,stroke-width:1px
    classDef ingestion fill:#cfe2f3,stroke:#0b5394,stroke-width:1px
    classDef pipeline fill:#d9ead3,stroke:#38761d,stroke-width:1px
    classDef datastore fill:#fce5cd,stroke:#e69138,stroke-width:1px
    classDef ui fill:#d9d2e9,stroke:#674ea7,stroke-width:1px
    classDef deployment fill:#cfe2f3,stroke:#134f5c,stroke-width:1px
```
