How bcastelino/sec-financial-chatbot works
Project type and purpose
• Type: Full-stack data-driven RAG (Retrieval-Augmented Generation) chatbot.
• Purpose: Download, parse, and index SEC 10-K filings (text + XBRL), then answer user queries via a Streamlit UI by combining vector similarity search with structured fact lookup and an LLM (OpenRouter’s Llama 3.3 Nemotron Super 49B).
File structure
• Top-level scripts and modules:
• extractor.py – pulls raw 10-K text & XBRL from SEC EDGAR
• ingestor.py – parses, chunks, and writes embeddings to ChromaDB
• retriever.py – encapsulates vector + structured XBRL retrieval logic
• analyzer.py – implements the RAG pipeline and LLM invocation
• streamlit_app.py + styles.css – front-end chat interface
• config.py – central configuration (companies, years, DB paths, model settings)
• Storage:
• data/filings – raw text filings
• data/xbrl – per-filing CSV exports of financial statements
• data/filings_metadata.json – metadata index
• chroma_db/ – persistent vector store (binary, SQLite, pickle)
• Miscellaneous:
• sec_filings_edgartools.ipynb – exploratory notebook
• .env.example, requirements.txt, LICENSE, README.md
README for additional insights
• Confirms a hybrid retrieval approach: semantic + structured XBRL
• Describes a four-stage pipeline: Extraction → Parsing → Vector Storage → RAG Chat
• Technologies: SentenceTransformers, ChromaDB, OpenRouter LLM, Streamlit
• Key config knobs: chunk size, overlap, number of chunks, model endpoints
a. Main components
1. External Services
– SEC EDGAR (data source)
– OpenRouter LLM API
2. Data Ingestion Layer
– extractor.py
– ingestor.py
3. Raw & Processed Data Stores
– File system: data/filings, data/xbrl, metadata JSON
– Vector DB: chroma_db/
4. Retrieval & Analysis Layer
– retriever.py (vector + XBRL queries)
– analyzer.py (RAG orchestration)
5. Presentation Layer
– streamlit_app.py (UI)
b. Relationships & data flow
1. extractor.py → (pull) → SEC EDGAR → (write) → data/filings + data/xbrl
2. ingestor.py → (read) → data/… → (chunk & embed) → ChromaDB
3. streamlit_app.py → (user query) → analyzer.py
4. analyzer.py → (calls) → retriever.py:
• retriever.py → (vector search) → ChromaDB
• retriever.py → (fact lookup) → data/xbrl
5. analyzer.py → (prompts) → OpenRouter API → (response) → analyzer.py → streamlit_app.py
c. Architectural patterns & principles
• Modular pipeline: clear separation between extraction, ingestion, retrieval, and presentation.
• Hybrid retrieval: vector similarity + structured data lookup.
• Config-driven design: all companies/years/DB paths in config.py.
• Stateless front end; stateful vector DB persists across runs.
d. Key technologies & libraries
• Python scripts (extractor, ingestor, retriever, analyzer)
• ChromaDB for vector storage
• SentenceTransformers embeddings
• OpenRouter LLM for generation
• Streamlit for UI
• JSON/CSV for intermediate data