How marsidmali/crypto-news-assistant works
Main Components
• User Interface: The Streamlit-powered frontend (src/app.py) that collects user queries and displays the output.
• Data Ingestion:
• Scraper Module (src/scraper.py) that automatically scrapes crypto news from TradingView.
• Article preprocessing and chunking (implied within the ingestion process for optimal retrieval).
• RAG Pipeline (src/rag.py): The core module that implements the Retrieval Augmented Generation. This involves:
• Hybrid Retrieval: Combining two retrieval strategies:
▪ Vector Search (e.g., FAISS with MiniLM embeddings).
▪ Keyword Search (BM25).
• Smart Reranking: Utilizing Cohere API to rerank search results contextually.
• Response Generation: Taking the retrieved context and feeding it into the language model (OpenAI or Ollama) to generate a response.
• Configuration & External Dependencies:
• config.json holds settings like API keys and model type selections.
• External APIs and services including TradingView (for data), OpenAI/Ollama (for generation), and Cohere (for ranking).
• Inside the RAG Pipeline, branch into two sub-components (or stages):
• Additionally, include a separate process flow starting from a “Data Ingestion / Scraper Module” that gathers crypto news from TradingView. This module should feed data into a “Data Store” or “News Repository” (even though not explicitly stored, representing the preprocessing & indexing steps), which then connects to the retrieval components.
Architectural Patterns and Technologies
• Modular separation of concerns by clearly distinguishing the scraping (data ingestion), retrieval (hybrid search), and generation (model invocation) stages.
• External dependency integrations:
• TradingView for news scraping.
• OpenAI or Ollama for language models.
• Cohere for smart reranking.
• That the use of a hybrid search (vector and keyword) is a key architectural decision that improves retrieval accuracy.
• Mention that configuration management (via config.json) allows for easy swapping of models or APIs.