# bingluen/newsspider architecture

> Architecture diagram of the GitHub repository bingluen/newsspider, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bingluen/newsspider
- Repository: https://github.com/bingluen/newsspider

## Overview

##### Project type and purpose
- This is an open-source, command-line news-scraping tool (“NewsSpider”) implemented as a single Python script.
- It fetches articles from two sources (自由時報/ltn and 中國時報/chinatimes), parses author info via DOM or regex, logs errors, and supports optional proxy configuration.

##### File structure
- Top-level files:
- newsSpider.py – the entire scraper, containing CLI parsing, HTTP requests, parsing logic, error handling and logging.
- readme.md – usage instructions, parsing details, proxy setup, error-handling notes.
- There are no additional modules or directories, so the architecture is flat but can be logically split into functional components.

##### README for additional insights
- Command-line parameters drive source selection and date ranges.
- Parsing logic differs by source:
- chinatimes – DOM extraction of `<article>` → `div.rp_name`
- ltn – regex on author strings (customizable via `parseAuthor`).
- Errors write both logs and failed URLs to a log file.
- Proxy settings are hard-coded around lines 53–56.

##### Main components
a. CLI Interface
– Parses `sys.argv` into `source`, `start_date`, `end_date`
b. Spider Controller
– Orchestrates date iteration and dispatches fetch tasks
c. HTTP Client
– Wraps `requests` or `urllib2` calls
– Applies proxy configuration if present
d. Parser Module
– Contains:
- DOM parser for chinatimes (e.g., BeautifulSoup)
- Regex parser for ltn (configurable patterns in `parseAuthor`)
– Extracts title, author, date, URL, content
e. Storage & Logging
– Writes successful items to output (console or file)
– On error: appends error message + URL to a log file
f. External Systems
– News websites (自由時報, 中國時報)
– Optional HTTP proxy

##### Relationships and interactions
- Controller → HTTP Client (fetch requests per date and page)
- HTTP Client → External Websites (GET requests, proxy optional)
- HTTP Client → Controller (HTML response)
- Controller → Parser Module (pass HTML chunk + source)
- Parser Module → Controller (returns structured data or parsing error)
- Controller → Storage & Logging (writes data or logs)
- Storage & Logging → File System (output files, log files)

##### Architectural patterns and principles
- Separation of concerns: CLI, network I/O, parsing, and logging are distinct logical units.
- Extensibility point: `parseAuthor` regex list can be extended for other patterns.
- Fail-fast logging: every HTTP or parsing failure is captured with URL context.
- Flat single-process architecture; no concurrency or persistence layers beyond the file system.

##### Technologies and libraries
- Python 2.x (note Unicode sensitivity on Windows)
- Standard libraries: `sys`, `logging`, `re`, `datetime`, `urllib2` or `requests`
- Third-party: `BeautifulSoup` (for DOM parsing)
- File I/O for logs/output

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Internal Modules (Blue)
    CLI["CLI Interface"]:::internal
    Controller["Spider Controller"]:::internal
    HTTPClient["HTTP Client"]:::internal
    Parser["Parser Module"]:::internal

    %% Storage & Logging (Orange)
    Storage["Storage & Logging"]:::storage
    FileSystem[(File System)]:::storage

    %% External Systems (Green)
    subgraph "External Systems"
        Websites["News Websites (ltn, chinatimes)"]:::external
        Proxy["Optional HTTP Proxy"]:::external
    end

    %% Data Flow
    CLI -->|"commands (source, dates)"| Controller
    Controller -->|"fetch request"| HTTPClient
    HTTPClient -->|"if proxy configured"| Proxy
    Proxy -->|"proxied GET request"| Websites
    HTTPClient -->|"direct GET request"| Websites
    Websites -->|"HTML response"| HTTPClient
    HTTPClient -->|"HTML response"| Controller
    Controller -->|"raw HTML + source"| Parser
    Parser -->|"parsed record / error"| Controller
    Controller -->|"successful data"| Storage
    Controller -->|"error log entry"| Storage
    Storage -->|"writes output & logs"| FileSystem

    %% Click Events
    click CLI "https://github.com/bingluen/newsspider/blob/master/newsSpider.py"
    click Controller "https://github.com/bingluen/newsspider/blob/master/newsSpider.py"
    click HTTPClient "https://github.com/bingluen/newsspider/blob/master/newsSpider.py"
    click Parser "https://github.com/bingluen/newsspider/blob/master/newsSpider.py"
    click Storage "https://github.com/bingluen/newsspider/blob/master/newsSpider.py"
    click Usage "https://github.com/bingluen/newsspider/blob/master/readme.md"

    %% Styles
    classDef internal fill:#D6EAF8,stroke:#3498DB,color:#1B4F72
    classDef external fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef storage fill:#FDEBD0,stroke:#D35400,color:#7E5109
```
