# marsidmali/crypto-news-assistant architecture

> Architecture diagram of the GitHub repository marsidmali/crypto-news-assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marsidmali/crypto-news-assistant
- Repository: https://github.com/marsidmali/crypto-news-assistant

## Overview

##### Main Components
- User Interface: The Streamlit-powered frontend (src/app.py) that collects user queries and displays the output.
- Data Ingestion:
- Scraper Module (src/scraper.py) that automatically scrapes crypto news from TradingView.
- Article preprocessing and chunking (implied within the ingestion process for optimal retrieval).
- RAG Pipeline (src/rag.py): The core module that implements the Retrieval Augmented Generation. This involves:
- Hybrid Retrieval: Combining two retrieval strategies:
▪ Vector Search (e.g., FAISS with MiniLM embeddings).
▪ Keyword Search (BM25).
- Smart Reranking: Utilizing Cohere API to rerank search results contextually.
- Response Generation: Taking the retrieved context and feeding it into the language model (OpenAI or Ollama) to generate a response.
- Configuration & External Dependencies:
- config.json holds settings like API keys and model type selections.
- External APIs and services including TradingView (for data), OpenAI/Ollama (for generation), and Cohere (for ranking).

- Inside the RAG Pipeline, branch into two sub-components (or stages):
- Additionally, include a separate process flow starting from a “Data Ingestion / Scraper Module” that gathers crypto news from TradingView. This module should feed data into a “Data Store” or “News Repository” (even though not explicitly stored, representing the preprocessing & indexing steps), which then connects to the retrieval components.

##### Architectural Patterns and Technologies
- Modular separation of concerns by clearly distinguishing the scraping (data ingestion), retrieval (hybrid search), and generation (model invocation) stages.
- External dependency integrations:
- TradingView for news scraping.
- OpenAI or Ollama for language models.
- Cohere for smart reranking.
- That the use of a hybrid search (vector and keyword) is a key architectural decision that improves retrieval accuracy.
- Mention that configuration management (via config.json) allows for easy swapping of models or APIs.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Level Components
    UI["UI – Streamlit Application"]:::ui
    UI -->|"UserQuery"| RAG["RAG Pipeline"]:::pipeline

    subgraph "Data Ingestion"
        SCRAPER["Scraper – TradingView News"]:::ingestion
        SCRAPER -->|"scrapes"| TRADINGVIEW["TradingView"]:::external
        SCRAPER --> NR["News Repository"]:::db
    end

    %% RAG Pipeline internal flow
    subgraph "RAG Pipeline Internal"
        RAG_CORE["RAG Pipeline Core"]:::pipeline
        RAG_CORE --> VS["Vector Search (FAISS+MiniLM)"]:::retrieval
        RAG_CORE --> KS["Keyword Search (BM25)"]:::retrieval
        VS --> HR["Hybrid Retrieval"]:::retrieval
        KS --> HR
        NR --> HR
        HR --> RR["Smart Reranking (Cohere)"]:::processing
        RR --> RG["Response Generation – OpenAI/Ollama"]:::processing
        RG --> LM["Language Model [OpenAI/Ollama]"]:::external
        RG -->|"ReturnsAnswer"| UI
        RR -->|"calls"| COHERE["Cohere API"]:::external
    end

    %% Styles
    classDef ui fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef ingestion fill:#a2d2ff,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#caffbf,stroke:#333,stroke-width:2px;
    classDef retrieval fill:#ffafcc,stroke:#333,stroke-width:2px;
    classDef processing fill:#bde0fe,stroke:#333,stroke-width:2px;
    classDef external fill:#ffc6ff,stroke:#333,stroke-width:2px;
    classDef db fill:#ffd6a5,stroke:#333,stroke-width:2px;

    %% Click Events
    click UI "https://github.com/marsidmali/crypto-news-assistant/blob/main/src/app.py"
    click SCRAPER "https://github.com/marsidmali/crypto-news-assistant/blob/main/src/scraper.py"
    click RAG "https://github.com/marsidmali/crypto-news-assistant/blob/main/src/rag.py"
```
