How techiepookie/crypto-insight-ai works
Main Components
• Data Acquisition: This component covers web scraping. Note that the tool fetches cryptocurrency-related articles from sites like thenewscrypto.com (using libraries such as requests and BeautifulSoup) and saves URLs into CSV files.
• PDF Conversion & Text Extraction: Articles are converted to PDFs (using pdfkit) and then text is extracted (via trafilatura) from these PDFs, filtering out noise like ads & navigation.
• Data Preprocessing: After extraction, the text is cleaned, tokenized, and normalized using NLP libraries like NLTK (including stemming and lemmatization). This ensures cryptocurrency-specific terms remain intact.
• Sentiment Analysis: Uses TextBlob to analyze and categorize the sentiment (positive, negative, neutral) of the extracted and processed text.
• Answer Retrieval: Implements the BM25 algorithm (via Rank-BM25) to index the processed text and, upon user query, deliver ranked answers enriched with additional details (source, score, sentiment).
• Data Storage & Artifacts: The project produces and manages several files (PDFs, cleaned text files, CSVs, and visualizations from matplotlib) that act as both intermediates and end-results.
Relationships and Interactions
• Data flows sequentially among these components:
● The scraping module (Data Acquisition) collects URLs and downloads article content.
● The PDF Conversion & Text Extraction module then converts these articles, extracting raw textual data.
● Next, the Data Preprocessing module cleans and prepares data for analysis.
● This sanitized text is passed to both the Sentiment Analysis module (for sentiment visualization and classification) and the Answer Retrieval module.
● The Answer Retrieval component uses the preprocessed text to build a BM25 index so that user queries can be answered by ranking relevant text passages.
• These modules interact in a pipeline fashion, where the output of one becomes the input of the next.
Relevant Architectural Patterns and Key Technologies
• The design follows a modular pipeline architecture where each step (scraping, extraction, processing, analysis, retrieval) is encapsulated as an independent component. This modularity facilitates easier testing, debugging, and potential future extensions.
• The project leverages a typical layered approach:
• Input Layer: Web scraping/data acquisition.
• Processing Layer: Includes conversion, extraction, and NLP preprocessing.
• Analysis Layer: For sentiment determination and ranking.
• Key technology stack includes Python libraries:
• requests and BeautifulSoup for scraping.
• pdfkit and trafilatura for PDF conversion and text extraction.
• NLTK and TextBlob for text processing and sentiment analysis.
• Rank-BM25 for the retrieval system.
• matplotlib for data visualization.
• Additionally, the use of external tools like wkhtmltopdf is essential for PDF conversion, making it part of the system’s external dependencies.
Additional Details
• The crypto_insight directory contains the primary scripts (e.g., crypto.py, crypto_scraping_model.py, process_and_push.py). Highlight this as the core implementation base.