How rodolflying/gpt_scraper works
Main Components
• API Scraper (api_scraper.py): This module uses the backend API method to retrieve conversation metadata and content by sending requests with the correct session headers. It relies on external configuration (the headers file) and uses the ‘requests’ library.
• Selenium Scraper (scraper.py): This component automates Chrome (using Selenium and ChromeDriver) to scrape the ChatGPT page. It simulates a browser environment to capture conversation history.
• Conversation Bot (conversation.py): This script enables interactive conversation with ChatGPT. It likely operates by reading the user’s input and then sending it to ChatGPT while managing a conversation session.
• Configuration/Headers (headers file): Although shown as a file named “headers”, this must contain the headers dictionaries required for authenticating the API calls. This acts as a configuration or credential provider.
• Dependencies/External Libraries: Note libraries like requests, selenium, beautifulsoup4, pandas, and python-dotenv, as well as external dependencies like ChromeDriver and Google Chrome. These are essential for the functioning of the scraping tools.
• (Implicit) Data Output: The README mentions that outputs are saved as JSON or CSV in an “outputs” folder, even if that folder isn’t explicitly shown in the file tree. This is where scraped data is stored.
Relationships and Interactions
• For the API Scraper:
• The user feeds the required headers (obtained via manual steps using Postman/Insomnia) into the headers file.
• api_scraper.py reads these headers and sends HTTP requests to specific ChatGPT endpoints (“conversations?offset=0&limit=20” and a conversation-specific endpoint).
• Scraped responses are processed and then output as JSON/CSV.
• For the Selenium Scraper:
• The script (scraper.py) automates Chrome (using Selenium) to simulate a logged-in session.
• It interacts with the live webpage (or browsing history) to retrieve similar conversation data.
• The data is then processed and stored in an “outputs” folder.
• For the Conversation Bot:
• The conversation.py module manages an interactive session.
• It listens for user input, then posts it (using one of the above approaches) and saves the ongoing conversation as a JSON file.
• It handles termination commands (‘exit’, ‘quit’, etc.) to properly close and persist the conversation.
Architectural Patterns & Design Principles
• Separation of Concerns: Different scripts handle specific methods to access ChatGPT (direct API vs. Selenium) and the interactive conversation. This separation lets each module focus on a single responsibility.
• Modularity: Each script (api_scraper.py, scraper.py, and conversation.py) is designed as an independent tool that can be updated or replaced as needed (for instance, the README mentions a future version using Selenium exclusively).
• Extensibility: The README also hints at adding future ML/IA functionality such as filtering, classifying, and calculating NLP insights.
• Data Flow: The clear directional data flow goes from the external ChatGPT service (accessed via browser requests or automated scraping) into the local processing scripts, and finally to the persistent “outputs” storage.
Diagrammatic Elements
○ API Scraper (api_scraper.py)
○ Selenium Scraper (scraper.py)
○ Conversation Bot (conversation.py)
○ From “Headers/Configuration” to API Scraper.
○ From API Scraper and Selenium Scraper to “ChatGPT Endpoint”.
○ From ChatGPT Endpoint back to the scrapers.
○ Modules (e.g., rectangles for scripts)