# bcastelino/sec-financial-chatbot architecture

> Architecture diagram of the GitHub repository bcastelino/sec-financial-chatbot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bcastelino/sec-financial-chatbot
- Repository: https://github.com/bcastelino/sec-financial-chatbot

## Overview

##### Project type and purpose
- Type: Full-stack data-driven RAG (Retrieval-Augmented Generation) chatbot.
- Purpose: Download, parse, and index SEC 10-K filings (text + XBRL), then answer user queries via a Streamlit UI by combining vector similarity search with structured fact lookup and an LLM (OpenRouter’s Llama 3.3 Nemotron Super 49B).

##### File structure
- Top-level scripts and modules:
- extractor.py – pulls raw 10-K text & XBRL from SEC EDGAR
- ingestor.py – parses, chunks, and writes embeddings to ChromaDB
- retriever.py – encapsulates vector + structured XBRL retrieval logic
- analyzer.py – implements the RAG pipeline and LLM invocation
- streamlit_app.py + styles.css – front-end chat interface
- config.py – central configuration (companies, years, DB paths, model settings)
- Storage:
- data/filings – raw text filings
- data/xbrl – per-filing CSV exports of financial statements
- data/filings_metadata.json – metadata index
- chroma_db/ – persistent vector store (binary, SQLite, pickle)
- Miscellaneous:
- sec_filings_edgartools.ipynb – exploratory notebook
- .env.example, requirements.txt, LICENSE, README.md

##### README for additional insights
- Confirms a hybrid retrieval approach: semantic + structured XBRL
- Describes a four-stage pipeline: Extraction → Parsing → Vector Storage → RAG Chat
- Technologies: SentenceTransformers, ChromaDB, OpenRouter LLM, Streamlit
- Key config knobs: chunk size, overlap, number of chunks, model endpoints

a. Main components
1. External Services
– SEC EDGAR (data source)
– OpenRouter LLM API
2. Data Ingestion Layer
– extractor.py
– ingestor.py
3. Raw & Processed Data Stores
– File system: data/filings, data/xbrl, metadata JSON
– Vector DB: chroma_db/
4. Retrieval & Analysis Layer
– retriever.py (vector + XBRL queries)
– analyzer.py (RAG orchestration)
5. Presentation Layer
– streamlit_app.py (UI)
b. Relationships & data flow
1. extractor.py → (pull) → SEC EDGAR → (write) → data/filings + data/xbrl
2. ingestor.py → (read) → data/… → (chunk & embed) → ChromaDB
3. streamlit_app.py → (user query) → analyzer.py
4. analyzer.py → (calls) → retriever.py:
- retriever.py → (vector search) → ChromaDB
- retriever.py → (fact lookup) → data/xbrl
5. analyzer.py → (prompts) → OpenRouter API → (response) → analyzer.py → streamlit_app.py
c. Architectural patterns & principles
- Modular pipeline: clear separation between extraction, ingestion, retrieval, and presentation.
- Hybrid retrieval: vector similarity + structured data lookup.
- Config-driven design: all companies/years/DB paths in config.py.
- Stateless front end; stateful vector DB persists across runs.
d. Key technologies & libraries
- Python scripts (extractor, ingestor, retriever, analyzer)
- ChromaDB for vector storage
- SentenceTransformers embeddings
- OpenRouter LLM for generation
- Streamlit for UI
- JSON/CSV for intermediate data

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    subgraph "External Services"
        SEC["SEC EDGAR"]:::ext
        LLM["OpenRouter LLM API"]:::ext
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        Extractor["Extractor\n(extractor.py)"]:::proc
        Ingestor["Ingestor\n(ingestor.py)"]:::proc
    end

    %% Storage
    subgraph "Storage"
        Raw["Raw Filings\n(data/filings/)"]:::storage
        XBRL["XBRL CSV Exports\n(data/xbrl/)"]:::storage
        Metadata["Filings Metadata\n(data/filings_metadata.json)"]:::storage
        Chroma["ChromaDB\n(chroma_db/)"]:::storage
    end

    %% Retrieval & Analysis Layer
    subgraph "Retrieval & Analysis"
        Retriever["Retriever\n(retriever.py)"]:::proc
        Analyzer["Analyzer\n(analyzer.py)"]:::proc
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        UI["Streamlit UI\n(streamlit_app.py)"]:::ui
        Styles["styles.css"]:::config
    end

    %% Configuration and Miscellaneous
    Config["config.py"]:::config
    Notebook["sec_filings_edgartools.ipynb"]:::config
    Env[".env.example"]:::config
    Req["requirements.txt"]:::config
    Readme["README.md"]:::config
    License["LICENSE"]:::config
    Report["Ingestion_report.txt"]:::config

    %% Data Flows
    SEC -->|pull filings| Extractor
    Extractor -->|writes text| Raw
    Extractor -->|writes XBRL CSV| XBRL
    Extractor -->|writes metadata| Metadata

    Raw -->|read| Ingestor
    XBRL -->|read| Ingestor
    Ingestor -->|embeds & stores| Chroma

    UI -->|user query| Analyzer
    Analyzer -->|calls| Retriever
    Retriever -->|vector search| Chroma
    Retriever -->|fact lookup| XBRL
    Analyzer -->|prompts| LLM
    LLM -->|response| Analyzer
    Analyzer -->|answers| UI

    %% Configuration Controls
    Config -.-> Extractor
    Config -.-> Ingestor
    Config -.-> Retriever
    Config -.-> Analyzer
    Config -.-> UI

    %% Miscellaneous Links
    Notebook -.-> Extractor
    Env -.-> Config
    Req -.-> Config
    Readme -.-> Config
    License -.-> Config
    Report -.-> Ingestor

    %% Click Events
    click Extractor "https://github.com/bcastelino/sec-financial-chatbot/blob/main/extractor.py"
    click Ingestor "https://github.com/bcastelino/sec-financial-chatbot/blob/main/ingestor.py"
    click Retriever "https://github.com/bcastelino/sec-financial-chatbot/blob/main/retriever.py"
    click Analyzer "https://github.com/bcastelino/sec-financial-chatbot/blob/main/analyzer.py"
    click UI "https://github.com/bcastelino/sec-financial-chatbot/blob/main/streamlit_app.py"
    click Styles "https://github.com/bcastelino/sec-financial-chatbot/blob/main/styles.css"
    click Config "https://github.com/bcastelino/sec-financial-chatbot/blob/main/config.py"
    click Raw "https://github.com/bcastelino/sec-financial-chatbot/tree/main/data/filings/"
    click XBRL "https://github.com/bcastelino/sec-financial-chatbot/tree/main/data/xbrl/"
    click Metadata "https://github.com/bcastelino/sec-financial-chatbot/blob/main/data/filings_metadata.json"
    click Chroma "https://github.com/bcastelino/sec-financial-chatbot/tree/main/chroma_db/"
    click Notebook "https://github.com/bcastelino/sec-financial-chatbot/blob/main/sec_filings_edgartools.ipynb"
    click Env "https://github.com/bcastelino/sec-financial-chatbot/blob/main/.env.example"
    click Req "https://github.com/bcastelino/sec-financial-chatbot/blob/main/requirements.txt"
    click Readme "https://github.com/bcastelino/sec-financial-chatbot/blob/main/README.md"
    click License "https://github.com/bcastelino/sec-financial-chatbot/tree/main/LICENSE"
    click Report "https://github.com/bcastelino/sec-financial-chatbot/blob/main/Ingestion_report.txt"

    %% Styles
    classDef storage fill:#cce5ff,stroke:#333
    classDef proc fill:#d4edda,stroke:#333
    classDef ext fill:#f8d7da,stroke:#333
    classDef ui fill:#f5c6cb,stroke:#333
    classDef config fill:#fff3cd,stroke:#333
```
