How amankumarverma11/twitter-trends-scraper works
Main Components
• Frontend: Represented by the React application built with Vite, which handles user interactions and displays the scraped Twitter trends.
• Backend: A Node.js and Express server that initiates scraping using Selenium WebDriver, communicates with Twitter, and orchestrates data retrieval.
• Database: MongoDB, which is used to store the scraped data.
• External Services:
• Twitter: The target source from where the trends are scraped.
• ProxyMesh: Used for managing proxies for web scraping.
• Environment Variables: Managed via dotenv, ensuring sensitive config data like credentials are kept secure.
Relationships and Interactions
• That the Backend interacts with external services:
• A connection from the Backend to Twitter (via Selenium for scraping).
• A connection from the Backend to ProxyMesh for proxy configuration.
Architectural Patterns & Principles
• Separation of Concerns: The design distinctly separates the frontend (user interface) from backend operations (scraping logic and data management).
• Modular Structure: Given the clear directory structure (frontend, backend, assets), emphasize how each module plays a specific role.
• API Layer: Highlight that the frontend and backend communicate through a defined RESTful API endpoint setup managed in the Node.js Express server.
• Integration of External Dependencies: Explicitly note the use of Selenium for automation, MongoDB for persistence, and ProxyMesh for robust scraping.
Technologies, Frameworks, and Libraries
• Frontend: React, Vite, Framer Motion (for animations)
• Backend: Node.js, Express, Selenium WebDriver
• Database: MongoDB
• Other: dotenv for environment management and ProxyMesh for proxy requests
• Clear labels for each component such as “Frontend (React/Vite)”, “Backend (Node.js/Express)”, “Database (MongoDB)”, “Scraping Service (Selenium)”, “External Service (ProxyMesh)”, and “Twitter”.
• User → Frontend → Backend → MongoDB,
• Backend → Twitter (scraping via Selenium) and → ProxyMesh (for handling proxy requests).
• Notes or annotations that highlight important architectural considerations like the use of environment variables via dotenv and modular code organization as indicated by the project’s file structure.