How bingluen/newsspider works
Project type and purpose
• This is an open-source, command-line news-scraping tool (“NewsSpider”) implemented as a single Python script.
• It fetches articles from two sources (自由時報/ltn and 中國時報/chinatimes), parses author info via DOM or regex, logs errors, and supports optional proxy configuration.
File structure
• Top-level files:
• newsSpider.py – the entire scraper, containing CLI parsing, HTTP requests, parsing logic, error handling and logging.
• readme.md – usage instructions, parsing details, proxy setup, error-handling notes.
• There are no additional modules or directories, so the architecture is flat but can be logically split into functional components.
README for additional insights
• Command-line parameters drive source selection and date ranges.
• Parsing logic differs by source:
• chinatimes – DOM extraction of <article> → div.rp_name
• ltn – regex on author strings (customizable via parseAuthor).
• Errors write both logs and failed URLs to a log file.
• Proxy settings are hard-coded around lines 53–56.
Main components
a. CLI Interface
– Parses sys.argv into source, start_date, end_date
b. Spider Controller
– Orchestrates date iteration and dispatches fetch tasks
c. HTTP Client
– Wraps requests or urllib2 calls
– Applies proxy configuration if present
d. Parser Module
– Contains:
• DOM parser for chinatimes (e.g., BeautifulSoup)
• Regex parser for ltn (configurable patterns in parseAuthor)
– Extracts title, author, date, URL, content
e. Storage & Logging
– Writes successful items to output (console or file)
– On error: appends error message + URL to a log file
f. External Systems
– News websites (自由時報, 中國時報)
– Optional HTTP proxy
Relationships and interactions
• Controller → HTTP Client (fetch requests per date and page)
• HTTP Client → External Websites (GET requests, proxy optional)
• HTTP Client → Controller (HTML response)
• Controller → Parser Module (pass HTML chunk + source)
• Parser Module → Controller (returns structured data or parsing error)
• Controller → Storage & Logging (writes data or logs)
• Storage & Logging → File System (output files, log files)
Architectural patterns and principles
• Separation of concerns: CLI, network I/O, parsing, and logging are distinct logical units.
• Extensibility point: parseAuthor regex list can be extended for other patterns.
• Fail-fast logging: every HTTP or parsing failure is captured with URL context.
• Flat single-process architecture; no concurrency or persistence layers beyond the file system.
Technologies and libraries
• Python 2.x (note Unicode sensitivity on Windows)
• Standard libraries: sys, logging, re, datetime, urllib2 or requests
• Third-party: BeautifulSoup (for DOM parsing)
• File I/O for logs/output