# vinay-gatech/stocks-insights-ai-agent architecture

> Architecture diagram of the GitHub repository vinay-gatech/stocks-insights-ai-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vinay-gatech/stocks-insights-ai-agent
- Repository: https://github.com/vinay-gatech/stocks-insights-ai-agent

## Overview

##### Project Type & Purpose
- This is a multi-faceted full-stack application that retrieves, processes, and analyzes stock-related news and financial data.
- It uses advanced AI workflows (Agentic RAG graphs) to interact with data, integrating asynchronous scraping, relational (PostgreSQL) and document (MongoDB) databases, and vector search (ChromaDB).

##### Analyzing the File Structure
- Key top-level folders include:
- config: Handles configuration loading.
- db: Contains database models and connection wrappers for MongoDB and PostgreSQL.
- rag_graphs: Contains multiple specialized RAG graphs for news, stock charts, and stock data – each implementing distinct chains and nodes for LLM-based generation and retrieval.
- rest_api: Exposes APIs for clients to interact with stock and news data.
- scraper: Implements asynchronous scraping for news and financial data.
- utils: Contains logging utilities.
- The file tree clearly shows modular separation by concerns – data ingestion, data persistence, processing workflows (RAG graphs), and API exposure.

##### High-Level Components & Relationships
a. Main Components:
- Data Ingestion Module (Scraper):
- Asynchronous scraper components in the scraper folder for fetching news and stock data.
- Data Persistence Layer:
- Two main databases: MongoDB (for news data) and PostgreSQL (for financial data).
- Model definitions in the db/models.
- RAG Workflow Engine:
- Three RAG graphs implemented under rag_graphs: News RAG Graph, Stock Data RAG Graph, and Stock Charts RAG Graph.
- Each graph consists of nodes (like retrieval, grading, web search; generate SQL, execute SQL, etc.) and chains built using LangChain, LangGraph, and LLM tooling.
- REST API Layer:
- REST API endpoints that provide various functionalities (price stats, charts, news by topic) are implemented in the rest_api directory.
- Configuration & Utility Modules:
- Config and utils directories for application settings and logging/tracing.
- Observability & Tracing:
- LangSmith integration to trace LLM calls across workflows.

b. Relationships and Data Flow:
- The Scraper fetches data asynchronously and stores news in MongoDB and financial data in PostgreSQL.
- The persistence layer then feeds into RAG graphs:
- The News RAG Graph first attempts a semantic search on stored news documents (possibly using ChromaDB as backing vector store) and, if needed, triggers additional web search steps.
- The Stock Data RAG Graph and Stock Charts RAG Graph generate SQL queries from user input, execute them on the PostgreSQL database, and use LLMs to generate readable results or chart data.
- The REST API acts as the entry point for client requests that trigger these workflows.
- Configurations (loaded in the config module) and utility modules (like logging) are integrated across all layers ensuring consistent behavior and observability.

##### Architectural Patterns & Technologies
- Microservice-like modular separation within a monorepo – each function (scraping, RAG processing, APIs) is decoupled.
- Utilizes asynchronous programming for data scraping.
- Follows a chain-of-responsibility style within RAG workflows (nodes process sequentially – retrieval, grading, search, generation).
- Employs external services and libraries such as LangChain, LangGraph, LCEL, and LangSmith for LLM interactions and observability.

##### Final Advice to the Principal Software Engineer
- Be very detailed: Label every component with its purpose (e.g., “News RAG Graph: Node - Web Search (LLM & TavilySearch)”).
- Use annotations to mention technologies (e.g., “ChromaDB for vector search”, “LangChain for chaining LLM calls”, “PostgreSQL for structured financial data”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        A["News Scraper"]:::ingestion
        B["Stock Data Scraper"]:::ingestion
    end

    %% REST API Layer
    subgraph "REST API Layer"
        H["REST API: Endpoints(/, /stock/{ticker}/price-stats, /stock/{ticker}/chart, /news/{ticker})"]:::api
    end

    %% RAG Workflow Engine
    subgraph "RAG Workflow Engine"
        E["News RAG Graph: retrieve,grade,search,generate"]:::rag
        F["Stock Data RAG Graph: generateSQL,executeSQL,generateResults"]:::rag
        G["Stock Charts RAG Graph: generateSQL,executeSQL"]:::rag
    end

    %% Data Persistence Layer
    subgraph "Data Persistence Layer"
        C["MongoDB (News Data)"]:::persistence
        D["PostgreSQL (Financial Data)"]:::persistence
    end

    %% Support Modules
    subgraph "Support Modules"
        I["Config Loader"]:::config
        J["Logger"]:::logging
        K["LangSmith (Observability)"]:::observability
    end

    %% Data Flow - Ingestion to Persistence
    A --> C
    B --> D

    %% Data Flow - REST API to RAG Workflows
    H --> E
    H --> F
    H --> G

    %% RAG Workflows accessing Persistence
    E --> C
    F --> D
    G --> D

    %% RAG Workflows integrated with Observability
    E --> K
    F --> K
    G --> K

    %% Config Loader linked to all main components
    I -.-> A
    I -.-> B
    I -.-> C
    I -.-> D
    I -.-> E
    I -.-> F
    I -.-> G
    I -.-> H

    %% Logger linked to all main components
    J -.-> A
    J -.-> B
    J -.-> C
    J -.-> D
    J -.-> E
    J -.-> F
    J -.-> G
    J -.-> H

    %% Click Events
    click I "https://github.com/vinay-gatech/stocks-insights-ai-agent/blob/main/config/config_loader.py"
    click A "https://github.com/vinay-gatech/stocks-insights-ai-agent/blob/main/scraper/news_scraper.py"
    click B "https://github.com/vinay-gatech/stocks-insights-ai-agent/blob/main/scraper/stock_data_scraper.py"
    click C "https://github.com/vinay-gatech/stocks-insights-ai-agent/blob/main/db/mongo_db.py"
    click D "https://github.com/vinay-gatech/stocks-insights-ai-agent/blob/main/db/postgres_db.py"
    click E "https://github.com/vinay-gatech/stocks-insights-ai-agent/tree/main/rag_graphs/news_rag_graph/"
    click F "https://github.com/vinay-gatech/stocks-insights-ai-agent/tree/main/rag_graphs/stock_data_rag_graph/"
    click G "https://github.com/vinay-gatech/stocks-insights-ai-agent/blob/main/rag_graphs/stock_charts_graph/main.py"
    click H "https://github.com/vinay-gatech/stocks-insights-ai-agent/blob/main/rest_api/main.py"
    click J "https://github.com/vinay-gatech/stocks-insights-ai-agent/blob/main/utils/logger.py"

    %% Class Definitions for Colors
    classDef ingestion fill:#D6EAF8,stroke:#5DADE2,stroke-width:2px;
    classDef persistence fill:#F9E79F,stroke:#F4D03F,stroke-width:2px;
    classDef rag fill:#FAD7A0,stroke:#F39C12,stroke-width:2px;
    classDef api fill:#D5F5E3,stroke:#58D68D,stroke-width:2px;
    classDef config fill:#F2D7D5,stroke:#E74C3C,stroke-width:2px;
    classDef logging fill:#E8DAEF,stroke:#AF7AC5,stroke-width:2px;
    classDef observability fill:#D1F2EB,stroke:#48C9B0,stroke-width:2px;
```
