How nit-in/pib works
This project is a web scraping tool built using the Scrapy framework, focused on downloading and processing Press Information Bureau (PIB) articles from India. The project provides not only the scraping mechanism through its spiders (found in pibindia/spiders) but also several command‐line interfaces (e.g. “pib”, “pib_min”, “pib_today”, etc.) to trigger various scraping workflows. In addition, there are GitHub Actions workflows defined in the .github directory that automate tasks such as daily scraping runs and packaging releases.
Main Components
• Scrapy Engine & Core Spider Package:
– Inside that, represent a “Spider” component (pibindia/spiders) that contains individual spider scripts (e.g. pib.py, pib_archives.py, pib_daily.py).
– Include components for “Middlewares” (pibindia/middlewares.py), “Pipelines” (pibindia/pipelines.py), “Items” (pibindia/items.py), and “Settings” (pibindia/settings.py). These illustrate how data is processed and configured within Scrapy.
• Command-Line Interface (CLI) Tools:
– Represent a CLI/Entry Point layer that triggers commands like “pib”, “pib_month”, “pib_min”, “pib_today”, etc. This serves as the user interface to start scraping jobs.
• Configuration & Environment:
– Illustrate configuration management using the .env file along with configuration files like scrapy.cfg and requirements.txt. These are essential for setting up the environment and launching the spiders.
• GitHub Actions & CI/CD Integration:
• External Data Sources & Outputs:
– Include an “Output / Release Packaging” component that covers the generated archive zips (PIB_Daily, PIB_Monthly, PIB_Text, PIB_Links) and their distribution via GitHub releases.
– Optionally, note the “ministry_code” functionality (using ministries.txt as a lookup) which is used in the pib_min workflow for filtering articles by ministry.
Relationships and Interactions
• The CLI triggers the Scrapy engine with appropriate start/end date parameters (or ministry code, etc.) to start a scraping job.
• The Spider components interact with the external PIB website to download HTML/PDF content.
• Extracted content flows from the Spider to Item Pipelines where processing (e.g., cleaning, filtering, formatting) occurs.
• Middlewares intercept requests/responses if additional modification or logging is required.
• Processed data is then stored locally and packaged, triggering GitHub Actions workflows (as per .github/workflows) for additional deployment or release steps.
• Configuration settings from .env and settings.py are used throughout the system to manage parameters (including tokens and environment secrets) required for scraping and CI tasks.
Architectural Patterns and Design Principles
• Use of Scrapy’s modular architecture (spiders, pipelines, middlewares) follows a pipeline processing pattern.
• Separation of concerns is clear: scraping logic (spiders) is decoupled from processing (pipelines) and configuration (settings and .env).
• CI/CD integration via GitHub Actions ensures that scraping tasks and release packaging run automatically without manual intervention.
Relevant Technologies and Frameworks
• Python & Scrapy: Core framework for scraping.
• GitHub Actions: For CI/CD automation.
• Environment management through .env and Python configuration.
• Shell scripting and command-line interfaces for local operation.