How goutham7675/internshala-automation works
Project Type and Purpose
• This is a backend automation tool written in Node.js. It uses Puppeteer to drive a headless browser, log in to Internshala, run searches, scrape internship/job data, and save results. There is no frontend or UI—just scripted automation.
File Structure
• Top-level scripts:
• index.js – the main orchestrator
• internship.js – a module for scraping internship listings
• cover.js – a module for composing cover letters or related tasks
• Configuration:
• package.json / package-lock.json – lists dependencies (Puppeteer, etc.)
• .gitattributes – repo settings
• .env (implicit) – environment variables for credentials
• Dependencies: node_modules folder holds Puppeteer and other npm packages.
README for Additional Insights
• Key features: automated login, parameterized searches, data extraction, configurable output (JSON/CSV).
• Tech stack: Node.js + Puppeteer.
• Extensibility: a scripts/ folder can house additional task modules.
Main Components
a. Configuration Layer
– Environment Variables (.env)
– package.json (dependency manifest)
b. Orchestrator
– index.js (initializes config, selects tasks)
c. Task Modules
– internship.js, cover.js, any scripts in scripts/
d. Puppeteer Engine
– Headless Chrome instance driven by the Puppeteer library
e. External Service
– Internshala website (login, search, data pages)
f. Storage Layer
– File System output (JSON/CSV)
Relationships and Data Flow
1. Startup: Node.js loads index.js → reads .env → loads config.
2. Task Invocation: index.js calls specific task module (e.g., internship.js).
3. Browser Automation: task module imports Puppeteer → launches headless browser → navigates to Internshala.
4. Data Exchange:
• Orchestrator → Task Module: “run internship scrape with filters X”
• Task Module → Puppeteer: “navigate to URL, login, extract data”
• Puppeteer ↔ External Site: HTTP requests, DOM interactions
• Task Module → Orchestrator: returns scraped data
• Orchestrator → Storage Layer: writes JSON/CSV
5. Shutdown: Puppeteer browser closes, process ends.
Architectural Patterns and Design Principles
• Modularity: clear separation between orchestrator and task scripts
• Stateless Execution: each run is fresh; no persistent server or DB
• Configuration-Driven: environment variables for credentials; script parameters for search filters
• Extensibility: add new scripts in a scripts/ folder to support additional tasks
Essential Details to Capture
• Environment variable injection at startup.
• That node_modules is a dependency bundle, not runtime logic.