How quinnyen/2024_ewantcrawler works
Project type and purpose
• This is a Python-based desktop data-scraping application with a GUI.
• Purpose: Automatically log into the “Ewant” course platform using Selenium, crawl course statistics (enrollment/pass counts), display progress in a UI, and export results to Excel.
File structure
• Top-level scripts & configs:
• main.py – application entry point
• build.py/.bat – build or packaging scripts
• requirements.txt – Python dependencies
• dependencies/dlls – external DLLs for packaging (e.g., cx_Freeze)
• Source code under src/:
• src/ui – UI layer (main_window.py)
• src/crawler – core crawling logic (login.py, parser.py, export.py)
• src/utils – utilities (config management, resource loading)
• No service-based or distributed modules—this is a single binary desktop application.
README for additional insights
• Uses Selenium to automate browser login and navigation.
• Outputs an Excel report.
• Reads/stores credentials via the OS keychain (handled in utils/config.py).
• UI allows keyword input, start/stop crawl, view logs, export report.
a. Main components
1) UI Component:
– Module: src/ui/main_window.py
– Technology: Tkinter or PyQt (inferred)
2) Crawler Component:
– Sub-components:
• Login Service (login.py)
• Parser Service (parser.py)
• Export Service (export.py)
– Technology: Selenium WebDriver
3) Utils Component:
– Config Manager (config.py) – handles keychain I/O
– Resource Loader (resource_utils.py)
4) External Systems:
– Ewant Web Platform (over HTTP/S via Selenium)
– OS Keychain (credential storage)
– File System (Excel output)
b. Relationships and interactions
• The UI triggers the Crawler Controller in src/crawler/export.py.
• Crawler Controller → Login Service → Selenium → Ewant platform
• On successful login → Parser Service navigates pages, collects data → returns data model
• UI listens for parser events and updates progress/log view
• When crawling finishes, UI invokes Export Service → writes .xlsx to File System
• Config Manager is invoked at startup to load saved credentials, and on exit to store new credentials.
c. Architectural patterns & design principles
• Clear separation of concerns: UI vs. business logic vs. utilities.
• Single-process, event-driven architecture: UI event loop drives crawling actions.
• Modular crawler services: each service class has one responsibility (login, parse, export).
d. Key technologies/frameworks
• Python 3.x runtime
• Selenium WebDriver (ChromeDriver, GeckoDriver, or similar)
• Excel export library (e.g., openpyxl or pandas)
• OS keychain interface (keyring library)
• GUI toolkit (Tkinter/PyQt)
Key points to capture
• Explicitly label modules with file names (e.g., src/crawler/login.py).
• Technology or library next to each component (e.g., “Selenium WebDriver”).
• Event flow (UI button click → crawl starts → progress events → export).
• One-way versus two-way interactions (e.g., UI polls parser events vs. parser pushes updates).