How v3rne42/news_crawler works
Project type and purpose
• This is a Node.js–based automated webcrawler and newsletter generator.
• It crawls configured news sites daily, extracts articles matching specified terms, summarizes content via the OpenAI API (GPT-4), then sends an email digest using the Resend API.
File structure
Top-level files and folders:
• main.js – entry point that orchestrates the pipeline
• config.json – runtime settings (sites, terms, schedule, API keys, email parameters)
• terminos.js – list of search terms and target site URLs
• text_analysis/topics_extractor.js – module that processes raw HTML/text to extract relevant topics
• browser/browserPath.js – helper to locate or configure the headless browser (Puppeteer or similar)
• assets/ – could contain email templates or static resources
• *.bat, *.ps1 – scripts to install startup tasks or reboot checks on Windows
• package.json – dependency definitions (OpenAI, Resend SDK, crawler libs)
Architectural clues:
• Modular pipeline: crawler → text analysis → summarizer → emailer
• Configuration-driven: all behavior tweaked via config.json and terminos.js
• Automation via OS task schedulers (batch scripts and PowerShell)
README for additional insights
• Confirms pipeline steps: crawl sites, filter by date and terms, generate summaries, then email.
• Spells out config.json fields (sender, recipients, schedule hour, OpenAI and Resend API keys, sensitivity, language).
• Mentions customization points (adding/removing terms or websites).