How bdelion/web-scraper works
The repository’s active product is a Node.js weather-scraping CLI. Its main workflow reads date ranges from Excel or accepts a custom range, resolves a station, fetches and formats observations, then writes Excel or prints JSON. README also describes scraping tutorials for other sites and Excel examples, represented as a separate exploratory branch. The map is based on the supplied source excerpts and tree; wiring inside unsampled exploratory files is not inferred.
Main entry points: meteo-scraping-file-input.js Command entry · weatherScraper.js Range retrieval · excelUtils.js Excel I/O · dateHourUtils.js Date conversions
Components 19
CLI workflow
- Command entry
src/meteo-scraping-file-input.js - Input date ranges
- Excel or JSON output
Weather scraping
- Range formatting
src/scrapers/weatherScraper.js - Station lookup
src/scrapers/weatherScraper.js - Range retrieval
src/scrapers/weatherScraper.js - Daily observations
src/scrapers/weatherScraper.js - Weather results
Date and file utilities
- Excel I/O
src/utils/excelUtils.js - Date conversions
src/utils/dateHourUtils.js - Date configuration
src/config/dayjsConfig.js - Temperature statistics
src/utils/mathUtils.js - Logging
src/utils/logUtils.js - Scraping errors
src/errors/customErrors.js
Exploratory examples
- Scraping tutorials
src/explorations
Other components
- CLI user
- Excel files
- Meteociel
- Tutorial target sites
Connections 17
- CLI user → Command entry: runs commands
- Command entry → Excel I/O: reads input
- Excel I/O → Excel files: reads or writes
- Command entry → Range formatting: formats ranges
- Command entry → Station lookup: resolves station
- Command entry → Range retrieval: fetches ranges
- Command entry → Date conversions: converts dates
- Date conversions → Date configuration: uses configuration
- Range retrieval → Daily observations: fetches days
- Station lookup → Meteociel: requests station ID
- Daily observations → Meteociel: requests observations
- Range retrieval → Weather results: returns weather data
- Daily observations → Weather results: produces observations
- Command entry → Excel I/O: writes results
- Command entry → Excel or JSON output: prints JSON
- Weather results → Excel or JSON output: supplies results
- Scraping tutorials → Tutorial target sites: demonstrate scraping