How skylaweber/newspaper4k works
Project type and purpose
• Newspaper4k is an open-source Python library and CLI tool for scraping, parsing, and curating news articles from arbitrary websites (and Google News).
• It provides a Python API (Article, Source, GoogleNewsSource classes), a command-line interface, multi-language support, multi-threaded downloads, HTML cleaning/parsing, NLP (keyword extraction, summarization), and output formatters (JSON, CSV, text).
File structure
• Top-level packages:
• newspaper/ – core library code (API, Article, Source, extractors, cleaners, parsers, network, nlp, utils, configuration)
• newspaper/extractors/ – specialized modules for title, body, images, metadata, etc.
• newspaper/languages/ – per-language tokenizers and stopword lists under resources/text.
• newspaper/resources/ – text resources (stopwords), site-source lists, user-agents.
• CLI entrypoint: newspaper/cli.py and newspaper/__main__.py
• Configuration: newspaper/configuration.py
• Concurrency: newspaper/mthreading.py uses ThreadPoolExecutor
• Google News integration: newspaper/google_news.py
• Output formatting: newspaper/outputformatters.py
• Build scripts & packaging: setup.py, pyproject.toml, requirements.txt
• Documentation (docs/) and tests (tests/) directories
README for additional insights
• Confirms key features: multi-threaded download, source/category detection, NLP summarization, Google News support, CLI usage, Python API usage
• Describes high-level workflow: build a Source, download articles in threads, parse via Article object, extract fields, output