How princeyyyy/tweets_scraper works
The Tweet Scraper project is a Python-based tool designed to fetch tweets using the Tweety library, enrich those tweets with additional details using the Scrapfly library, and then output the aggregated data into a CSV file. Even though the project is simple, we can still model a clear, modular architecture that highlights the major components and their interactions.
Main Components
• Authentication/Configuration Component: This encapsulates the storage and management of the Twitter account credentials for Tweety and the API key for Scrapfly. Although this isn’t a separate file, it is an essential part of the configuration within tweets_stream.py.
• Tweet Fetching Component: Using the Tweety library, this part handles the connection to the Twitter API and retrieves tweets from a specified user.
• Data Scraping Component: Once tweets are fetched, this module interacts with the Scrapfly API to scrape additional details for each tweet.
• Data Aggregation & Output Component: This gathers and possibly formats the fetched and scraped data and then writes the resulting dataset into a CSV file (tweet_details.csv).
• Logging/Monitoring Component: The script uses the loguru module for logging purposes to track the process flow and potential errors.
Relationships/Interactions
• Both components feed into the Data Aggregation & Output component, culminating in writing the output to a CSV.
• The Logging/Monitoring component should be portrayed as an overlay or parallel component that logs events from each step.
Architectural Patterns/Design Principles
• Modularity: Each major functionality (fetching, scraping, and data writing) is conceptually separated so that scaling or updating one part (e.g., switching to another API or output format) can be done with minimal interference.
• Sequential Pipeline: The process has a clear, linear data pipeline—authentication/configuration → fetching → scraping → output.
Technologies/Frameworks
• Python makes up the foundation.
• Tweety (for Twitter API interactions).
• Scrapfly (for web scraping additional tweet details).
• Loguru (for logging).
• The built-in csv module for handling CSV file operations.
• Asyncio and additional modules like typing, json, and jmespath serve as support tools for asynchronous operations and data processing.