# techiepookie/crypto-insight-ai architecture

> Architecture diagram of the GitHub repository techiepookie/crypto-insight-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/techiepookie/crypto-insight-ai
- Repository: https://github.com/techiepookie/crypto-insight-ai

## Overview

##### Main Components
- Data Acquisition: This component covers web scraping. Note that the tool fetches cryptocurrency-related articles from sites like thenewscrypto.com (using libraries such as requests and BeautifulSoup) and saves URLs into CSV files.
- PDF Conversion & Text Extraction: Articles are converted to PDFs (using pdfkit) and then text is extracted (via trafilatura) from these PDFs, filtering out noise like ads & navigation.
- Data Preprocessing: After extraction, the text is cleaned, tokenized, and normalized using NLP libraries like NLTK (including stemming and lemmatization). This ensures cryptocurrency-specific terms remain intact.
- Sentiment Analysis: Uses TextBlob to analyze and categorize the sentiment (positive, negative, neutral) of the extracted and processed text.
- Answer Retrieval: Implements the BM25 algorithm (via Rank-BM25) to index the processed text and, upon user query, deliver ranked answers enriched with additional details (source, score, sentiment).
- Data Storage & Artifacts: The project produces and manages several files (PDFs, cleaned text files, CSVs, and visualizations from matplotlib) that act as both intermediates and end-results.

##### Relationships and Interactions
- Data flows sequentially among these components:
● The scraping module (Data Acquisition) collects URLs and downloads article content.
● The PDF Conversion & Text Extraction module then converts these articles, extracting raw textual data.
● Next, the Data Preprocessing module cleans and prepares data for analysis.
● This sanitized text is passed to both the Sentiment Analysis module (for sentiment visualization and classification) and the Answer Retrieval module.
● The Answer Retrieval component uses the preprocessed text to build a BM25 index so that user queries can be answered by ranking relevant text passages.
- These modules interact in a pipeline fashion, where the output of one becomes the input of the next.

##### Relevant Architectural Patterns and Key Technologies
- The design follows a modular pipeline architecture where each step (scraping, extraction, processing, analysis, retrieval) is encapsulated as an independent component. This modularity facilitates easier testing, debugging, and potential future extensions.
- The project leverages a typical layered approach:
- Input Layer: Web scraping/data acquisition.
- Processing Layer: Includes conversion, extraction, and NLP preprocessing.
- Analysis Layer: For sentiment determination and ranking.
- Key technology stack includes Python libraries:
- requests and BeautifulSoup for scraping.
- pdfkit and trafilatura for PDF conversion and text extraction.
- NLTK and TextBlob for text processing and sentiment analysis.
- Rank-BM25 for the retrieval system.
- matplotlib for data visualization.
- Additionally, the use of external tools like wkhtmltopdf is essential for PDF conversion, making it part of the system’s external dependencies.

##### Additional Details
- The crypto_insight directory contains the primary scripts (e.g., crypto.py, crypto_scraping_model.py, process_and_push.py). Highlight this as the core implementation base.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Input Layer"
        A["Web Scraper"]:::input
    end

    subgraph "Processing Layer"
        B["PDF Converter & Text Extractor"]:::processing
        C["Data Preprocessor"]:::processing
        D["Sentiment Analyzer"]:::processing
    end

    subgraph "Analysis Layer"
        E["BM25 Retrieval Engine"]:::analysis
    end

    subgraph "Data Storage & Artifacts"
        F1["PDFs & Artifacts"]:::storage
        F2["Extracted Text"]:::storage
    end

    subgraph "External Dependencies"
        X["wkhtmltopdf"]:::external
    end

    A -->|"fetchesArticles"| B
    B -->|"convertsExtracts"| C
    C -->|"cleanedText"| D
    C -->|"cleanedText"| E
    D -->|"sentimentOutput"| F2
    E -->|"retrievalResults"| F2
    B -->|"createsPDFs"| F1
    B -->|"requires"| X

    click A "https://github.com/techiepookie/crypto-insight-ai/blob/main/crypto_insight/crypto_scraping_model.py"
    click B "https://github.com/techiepookie/crypto-insight-ai/blob/main/crypto_insight/crypto.py"
    click C "https://github.com/techiepookie/crypto-insight-ai/blob/main/crypto_insight/process_and_push.py"
    click D "https://github.com/techiepookie/crypto-insight-ai/blob/main/crypto_insight/crypto.py"
    click E "https://github.com/techiepookie/crypto-insight-ai/blob/main/crypto_insight/model.py"
    click F1 "https://github.com/techiepookie/crypto-insight-ai/tree/main/crypto_insight/"
    click F2 "https://github.com/techiepookie/crypto-insight-ai/blob/main/DataFiles/extracted_text.txt"

    classDef input fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef processing fill:#aed6f1,stroke:#333,stroke-width:2px;
    classDef analysis fill:#a9dfbf,stroke:#333,stroke-width:2px;
    classDef storage fill:#f5cba7,stroke:#333,stroke-width:2px;
    classDef external fill:#e6b0aa,stroke:#333,stroke-width:2px;
```
