How vzepec/scraper-challenge works
1. Purpose and boundary
- This is a Node.js/TypeScript web scraper for the OEFA Tribunal de Fiscalización Ambiental repository.
- It does not use a browser automation stack; instead it talks directly to the site’s JSF/PrimeFaces HTTP endpoints and reconstructs the hidden form submissions the UI would normally make.
2. Top-level flow
- The scraper starts with an initial GET to establish a session cookie (JSESSIONID) and capture the first ViewState.
- It then performs a search, paginates through the full result set, parses each page into records, downloads the Excel export once, and downloads each row’s PDF.
- Failures from PDF downloads are accumulated and can be retried later in a separate pass.
3. Entry and orchestration
- src/index.ts is the entrypoint; src/scraper/pipeline.ts is the main orchestrator.
- The pipeline coordinates all phases: session bootstrap, search, pagination, parsing, exports, PDF downloads, persistence, and retry handling.
- This is the central place to draw the “business flow” in an architecture diagram.
4. HTTP transport layer
- src/http/client.ts owns the actual HTTP client and session-aware requests.
- src/http/rateLimiter.ts constrains request pacing/concurrency so the scraper does not overwhelm the OEFA site.
- src/http/retry.ts implements retry/backoff behavior, especially for 429 Too Many Requests responses.
- This layer is the boundary between scraper logic and the external service.
5. JSF/PrimeFaces compatibility layer
- src/jsf/viewState.ts extracts and tracks the rotating ViewState token required by JSF.
- src/jsf/partialResponse.ts unwraps JSF partial responses, which are not plain HTML but XML envelopes containing the actual content.
- src/jsf/formParams.ts builds the POST bodies needed for search, pagination, export, and file download actions.
- These modules isolate all site-specific protocol quirks.
6. Scraping and extraction layer
- src/scraper/searchClient.ts handles the search and pagination lifecycle, including moving through pages by offset.
- src/scraper/recordParser.ts converts table rows into structured record objects.
- src/scraper/excelDownloader.ts triggers and saves the site’s full Excel export.
- src/scraper/pdfDownloader.ts downloads each document PDF and validates the result before persisting it.
7. Persistence layer
- src/persistence/recordStore.ts writes extracted records to JSON output.
- src/persistence/failureLog.ts records documents that could not be downloaded so they can be retried later.
- src/persistence/fileNaming.ts centralizes output naming, especially for PDFs and exported files, so filenames remain stable and deduplicated.
- These modules define the filesystem boundary of the application.
8. Shared configuration and types
- src/config.ts holds runtime configuration such as endpoints, delays, output locations, and other scraper parameters.
- src/types.ts contains shared data models used across HTTP, parsing, scraping, and persistence.
- This keeps the rest of the code strongly typed and avoids scattering constants or shape definitions.
9. Utility support
- src/utils/logger.ts provides structured logging for progress, warnings, and errors.
- src/utils/fs.ts handles directory creation and filesystem setup.
- src/utils/sleep.ts supports backoff and pacing between requests.
- These utilities are small, generic support services used across the pipeline.
10. Retry mode as a separate recovery path
- The repo’s retry flow re-establishes a fresh JSF session, navigates directly to the page offsets recorded in failures.json, and retries only the failed PDFs for those pages.
- This is important architecturally: retry is not just “repeat the same request,” but a stateful recovery path that must restore the correct page/view state before replaying downloads.
- The result is a two-phase system: primary crawl plus targeted recovery.
11. External dependency and protocol constraints
- The only real external system is the OEFA repository site itself, which dictates the architecture because of JSF statefulness, partial responses, and request throttling.
- The code structure is shaped around those constraints: session management, ViewState rotation, paginated search, export endpoints, and per-record download actions.
12. Repository shape and maintainability
- The project is intentionally layered: transport, JSF protocol handling, scraping orchestration, persistence, and utilities.
- That separation makes the architecture diagram straightforward: a single orchestrator in the middle, with well-defined dependencies outward to HTTP/JSF helpers and filesystem persistence.
- The repo also includes documentation (docs/SOLUTION_DESIGN.md, docs/SITE_NOTES.md) that records the site contract and design decisions, reinforcing that most complexity is in protocol adaptation rather than general crawling logic.
Main entry points: pipeline.ts Pipeline · client.ts HTTP client · searchClient.ts Search client · viewState.ts ViewState
Components 20
Entry & orchestration
Program entrypoint and the main crawl/retry pipeline.
HTTP & JSF protocol
Session-aware transport plus JSF/PrimeFaces compatibility helpers.
- HTTP client · Session-aware HTTP requests to the OEFA site.
src/http/client.ts - Rate limiter · Controls pacing and concurrency to avoid overwhelming the site.
src/http/rateLimiter.ts - Retry logic · Retries transient HTTP failures, especially 429 responses.
src/http/retry.ts - ViewState · Tracks the rotating JSF ViewState value.
src/jsf/viewState.ts - Partial response · Unwraps JSF XML partial responses into usable content.
src/jsf/partialResponse.ts - Form params · Builds POST bodies for search, paging, export, and downloads.
src/jsf/formParams.ts
Scraping stages
Search, parsing, export, and document download steps.
- Search client · Runs search and paginates through the result set.
src/scraper/searchClient.ts - Record parser · Converts table rows into structured records.
src/scraper/recordParser.ts - Excel export · Triggers and saves the full Excel export.
src/scraper/excelDownloader.ts - PDF downloader · Downloads and validates per-record PDFs.
src/scraper/pdfDownloader.ts
Persistence
Filesystem outputs and retry state.
- Record store · Writes extracted records to JSON files.
src/persistence/recordStore.ts - Failure log · Stores PDF download failures for later replay.
src/persistence/failureLog.ts - File naming · Centralizes stable names for PDFs and exports.
src/persistence/fileNaming.ts
Shared support
Configuration, types, and utilities used across the scraper.
- Config · Holds endpoints, delays, paths, and scraper parameters.
src/config.ts - Types · Shared data models used across the application.
src/types.ts - Logger · Structured logging for progress, warnings, and errors.
src/utils/logger.ts - Filesystem · Directory creation and setup helpers.
src/utils/fs.ts - Sleep · Pacing and backoff delay helper.
src/utils/sleep.ts
Connections 29
- index.ts → Pipeline: starts · Entrypoint launches the main pipeline.
- Pipeline → Config: reads config · Pipeline uses runtime settings for endpoints, pacing, and output paths.
- Pipeline → Logger: logs · Pipeline reports progress and failures.
- Pipeline → Filesystem: prepares output · Pipeline ensures output directories exist.
- Pipeline → HTTP client: requests · Pipeline performs session-aware HTTP calls through the client.
- Pipeline → Search client: drives crawl · Pipeline delegates search and pagination flow.
- Pipeline → Excel export: exports · Pipeline triggers the Excel export phase.
- Pipeline → PDF downloader: downloads · Pipeline initiates per-record PDF fetches.
- Pipeline → Record store: persists records · Pipeline writes extracted records to disk.
- Pipeline → Failure log: records failures · Pipeline stores failed PDF downloads for retry.
- Pipeline → File naming: names files · Pipeline relies on stable output naming.
- Pipeline → Sleep: paces work · Pipeline uses sleep for delays and backoff timing.
- HTTP client → Rate limiter: throttled by · HTTP client is constrained by rate limiting.
- HTTP client → Retry logic: retried by · HTTP client operations use retry/backoff behavior.
- HTTP client → ViewState: tracks · HTTP client works with rotating JSF ViewState.
- HTTP client → Partial response: unwraps · HTTP responses may need partial-response extraction.
- HTTP client → Config: uses config · Client needs endpoints and timeouts.
- Search client → ViewState: updates viewstate · Search and paging must keep the JSF token current.
- Search client → Form params: builds actions · Search client creates request bodies for search and paging.
- Search client → Record parser: parses rows · Search pages are converted into records.
- Excel export → Form params: builds export · Excel export uses JSF form parameters.
- PDF downloader → Form params: builds download · PDF fetches use JSF action parameters.
- PDF downloader → File naming: names outputs · Downloader uses naming helpers for saved PDFs.
- Record store → Types: serializes models · Record persistence writes shared record shapes.
- Failure log → Types: stores state · Failure log persists failed download metadata.
- File naming → Filesystem: writes paths · Naming helpers are used when creating filesystem outputs.
- ViewState → Partial response: interprets · ViewState handling depends on partial-response extraction.
- Form params → ViewState: embeds token · Form bodies include the current ViewState.
- Config → Types: defines shapes · Config and types establish shared runtime contracts.