# v3rne42/news_crawler architecture

> Architecture diagram of the GitHub repository v3rne42/news_crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/v3rne42/news_crawler
- Repository: https://github.com/v3rne42/news_crawler

## Overview

##### Project type and purpose
- This is a Node.js–based automated webcrawler and newsletter generator.
- It crawls configured news sites daily, extracts articles matching specified terms, summarizes content via the OpenAI API (GPT-4), then sends an email digest using the Resend API.

##### File structure
Top-level files and folders:
- main.js – entry point that orchestrates the pipeline
- config.json – runtime settings (sites, terms, schedule, API keys, email parameters)
- terminos.js – list of search terms and target site URLs
- text_analysis/topics_extractor.js – module that processes raw HTML/text to extract relevant topics
- browser/browserPath.js – helper to locate or configure the headless browser (Puppeteer or similar)
- assets/ – could contain email templates or static resources
- *.bat, *.ps1 – scripts to install startup tasks or reboot checks on Windows
- package.json – dependency definitions (OpenAI, Resend SDK, crawler libs)

Architectural clues:
- Modular pipeline: crawler → text analysis → summarizer → emailer
- Configuration-driven: all behavior tweaked via config.json and terminos.js
- Automation via OS task schedulers (batch scripts and PowerShell)

##### README for additional insights
- Confirms pipeline steps: crawl sites, filter by date and terms, generate summaries, then email.
- Spells out config.json fields (sender, recipients, schedule hour, OpenAI and Resend API keys, sensitivity, language).
- Mentions customization points (adding/removing terms or websites).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Scheduler / Orchestrator
    subgraph "Scheduler / Orchestrator"
        auto_start["automatic_startup.bat"]:::internal
        reboot_tasks["reboot_tasks_creation.bat"]:::internal
        check_reboot["check_and_reboot.ps1"]:::internal
        main_js["main.js"]:::internal
    end

    %% Configuration Store
    subgraph "Configuration Store"
        config_json["config.json"]:::config
        terminos["terminos.js"]:::config
    end

    %% Assets / Templates
    assets["assets/"]:::config

    %% Pipeline Modules
    subgraph "Pipeline"
        crawler["Webcrawler Module\n(browserPath.js)"]:::internal
        analyzer["Text Analysis Module\n(topics_extractor.js)"]:::internal
        summarizer["Summarization Module\n(OpenAI GPT-4)"]:::internal
        emailer["Email Module\n(Resend API)"]:::internal
    end

    %% Logging / Persistence
    logs["Logs / Output"]:::datastore

    %% External Services
    subgraph "External Services"
        news_websites["News Websites"]:::external
        openai_api["OpenAI API"]:::external
        resend_api["Resend API"]:::external
    end

    %% Data Flows
    auto_start --> main_js
    reboot_tasks --> main_js
    check_reboot --> main_js

    config_json --> main_js
    terminos --> main_js

    main_js --> crawler
    crawler -->|HTML| analyzer
    analyzer -->|Filtered Text| summarizer
    summarizer -->|Prompt| openai_api
    openai_api -->|Summary| summarizer
    summarizer --> emailer
    emailer -->|Email Payload| resend_api

    news_websites -->|News HTML| crawler
    assets --> emailer

    %% Logging at each stage
    main_js --> logs
    crawler --> logs
    analyzer --> logs
    summarizer --> logs
    emailer --> logs

    %% Click Events
    click main_js "https://github.com/v3rne42/news_crawler/blob/main/main.js"
    click auto_start "https://github.com/v3rne42/news_crawler/blob/main/automatic_startup.bat"
    click reboot_tasks "https://github.com/v3rne42/news_crawler/blob/main/reboot_tasks_creation.bat"
    click check_reboot "https://github.com/v3rne42/news_crawler/blob/main/check_and_reboot.ps1"
    click browserPath "https://github.com/v3rne42/news_crawler/blob/main/browser/browserPath.js"
    click topics_extractor "https://github.com/v3rne42/news_crawler/blob/main/text_analysis/topics_extractor.js"
    click package_json "https://github.com/v3rne42/news_crawler/blob/main/package.json"
    click config_json "https://github.com/v3rne42/news_crawler/blob/main/config.json"
    click terminos "https://github.com/v3rne42/news_crawler/blob/main/terminos.js"
    click assets "https://github.com/v3rne42/news_crawler/tree/main/assets/"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef config fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef datastore fill:#D3D3D3,stroke:#333,stroke-width:1px
```
