# nit-in/pib architecture

> Architecture diagram of the GitHub repository nit-in/pib, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nit-in/pib
- Repository: https://github.com/nit-in/pib

## Overview

This project is a web scraping tool built using the Scrapy framework, focused on downloading and processing Press Information Bureau (PIB) articles from India. The project provides not only the scraping mechanism through its spiders (found in pibindia/spiders) but also several command‐line interfaces (e.g. “pib”, “pib_min”, “pib_today”, etc.) to trigger various scraping workflows. In addition, there are GitHub Actions workflows defined in the .github directory that automate tasks such as daily scraping runs and packaging releases.

##### Main Components
- Scrapy Engine & Core Spider Package:
– Inside that, represent a “Spider” component (pibindia/spiders) that contains individual spider scripts (e.g. pib.py, pib_archives.py, pib_daily.py).
– Include components for “Middlewares” (pibindia/middlewares.py), “Pipelines” (pibindia/pipelines.py), “Items” (pibindia/items.py), and “Settings” (pibindia/settings.py). These illustrate how data is processed and configured within Scrapy.
- Command-Line Interface (CLI) Tools:
– Represent a CLI/Entry Point layer that triggers commands like “pib”, “pib_month”, “pib_min”, “pib_today”, etc. This serves as the user interface to start scraping jobs.
- Configuration & Environment:
– Illustrate configuration management using the .env file along with configuration files like scrapy.cfg and requirements.txt. These are essential for setting up the environment and launching the spiders.
- GitHub Actions & CI/CD Integration:
- External Data Sources & Outputs:
– Include an “Output / Release Packaging” component that covers the generated archive zips (PIB_Daily, PIB_Monthly, PIB_Text, PIB_Links) and their distribution via GitHub releases.
– Optionally, note the “ministry_code” functionality (using ministries.txt as a lookup) which is used in the pib_min workflow for filtering articles by ministry.

##### Relationships and Interactions
- The CLI triggers the Scrapy engine with appropriate start/end date parameters (or ministry code, etc.) to start a scraping job.
- The Spider components interact with the external PIB website to download HTML/PDF content.
- Extracted content flows from the Spider to Item Pipelines where processing (e.g., cleaning, filtering, formatting) occurs.
- Middlewares intercept requests/responses if additional modification or logging is required.
- Processed data is then stored locally and packaged, triggering GitHub Actions workflows (as per .github/workflows) for additional deployment or release steps.
- Configuration settings from .env and settings.py are used throughout the system to manage parameters (including tokens and environment secrets) required for scraping and CI tasks.

##### Architectural Patterns and Design Principles
- Use of Scrapy’s modular architecture (spiders, pipelines, middlewares) follows a pipeline processing pattern.
- Separation of concerns is clear: scraping logic (spiders) is decoupled from processing (pipelines) and configuration (settings and .env).
- CI/CD integration via GitHub Actions ensures that scraping tasks and release packaging run automatically without manual intervention.

##### Relevant Technologies and Frameworks
- Python & Scrapy: Core framework for scraping.
- GitHub Actions: For CI/CD automation.
- Environment management through .env and Python configuration.
- Shell scripting and command-line interfaces for local operation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & User Interface
    subgraph "CLI & User Interface"
        CLI["CLI Commands"]:::cli
    end

    %% Configuration & Environment
    subgraph "Configuration & Environment"
        ENV[".env"]:::config
        SCRAPY_CFG["scrapy.cfg"]:::config
        REQ["requirements.txt"]:::config
    end

    %% Scrapy Engine & Core Spider Package
    subgraph "Scrapy Engine"
        SE["Scrapy Engine"]:::engine
        subgraph "Components"
            Spiders["Spiders (pibindia/spiders)"]:::engine
            Middlewares["Middlewares (pibindia/middlewares.py)"]:::engine
            Pipelines["Pipelines (pibindia/pipelines.py)"]:::engine
            Items["Items (pibindia/items.py)"]:::engine
            Settings["Settings (pibindia/settings.py)"]:::engine
        end
    end

    %% External Data Sources & Outputs
    subgraph "External & Output"
        PIB["PIB Website\n(https://www.pib.gov.in)"]:::external
        Output["Output/Release Packaging"]:::external
        Ministry["Ministry Code Lookup\n(ministries.txt)"]:::external
    end

    %% GitHub Actions & CI/CD Integration
    subgraph "CI/CD Integration"
        GitHub["GitHub Actions Workflows\n(.github/workflows)"]:::ci
        Dependabot["CI Config\n(.github/dependabot.yml)"]:::ci
    end

    %% Relationships & Data Flow
    CLI -->|"triggers"| SE
    CLI -->|"filters"| Ministry
    ENV -->|"configures"| CLI
    ENV -->|"configures"| SE
    SCRAPY_CFG --- ENV
    REQ --- ENV
    Spiders -->|"fetches"| PIB
    Spiders -->|"sends_data"| Pipelines
    Pipelines -->|"processes"| Output
    Output -->|"triggers"| GitHub
    GitHub --- Dependabot

    %% Styles
    classDef cli fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef config fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef engine fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef external fill:#F5CBA7,stroke:#B9770E,stroke-width:2px;
    classDef ci fill:#D2B4DE,stroke:#7D3C98,stroke-width:2px;

    %% Click Events
    click Spiders "https://github.com/nit-in/pib/tree/master/pibindia/spiders/"
    click Middlewares "https://github.com/nit-in/pib/blob/master/pibindia/middlewares.py"
    click Pipelines "https://github.com/nit-in/pib/blob/master/pibindia/pipelines.py"
    click Items "https://github.com/nit-in/pib/blob/master/pibindia/items.py"
    click Settings "https://github.com/nit-in/pib/blob/master/pibindia/settings.py"
    click ENV "https://github.com/nit-in/pib/blob/master/.env"
    click SCRAPY_CFG "https://github.com/nit-in/pib/blob/master/scrapy.cfg"
    click REQ "https://github.com/nit-in/pib/blob/master/requirements.txt"
    click GitHub "https://github.com/nit-in/pib/tree/master/.github/workflows/"
    click Dependabot "https://github.com/nit-in/pib/blob/master/.github/dependabot.yml"
    click Ministry "https://github.com/nit-in/pib/blob/master/ministries.txt"
```
