# vzepec/scraper-challenge architecture

> Architecture diagram of the GitHub repository vzepec/scraper-challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vzepec/scraper-challenge
- Repository: https://github.com/vzepec/scraper-challenge

## Overview

1. **Purpose and boundary**
   - This is a **Node.js/TypeScript web scraper** for the OEFA Tribunal de Fiscalización Ambiental repository.
   - It does not use a browser automation stack; instead it talks directly to the site’s **JSF/PrimeFaces** HTTP endpoints and reconstructs the hidden form submissions the UI would normally make.

2. **Top-level flow**
   - The scraper starts with an initial GET to establish a **session cookie (`JSESSIONID`)** and capture the first **`ViewState`**.
   - It then performs a search, paginates through the full result set, parses each page into records, downloads the Excel export once, and downloads each row’s PDF.
   - Failures from PDF downloads are accumulated and can be retried later in a separate pass.

3. **Entry and orchestration**
   - `src/index.ts` is the entrypoint; `src/scraper/pipeline.ts` is the main orchestrator.
   - The pipeline coordinates all phases: session bootstrap, search, pagination, parsing, exports, PDF downloads, persistence, and retry handling.
   - This is the central place to draw the “business flow” in an architecture diagram.

4. **HTTP transport layer**
   - `src/http/client.ts` owns the actual HTTP client and session-aware requests.
   - `src/http/rateLimiter.ts` constrains request pacing/concurrency so the scraper does not overwhelm the OEFA site.
   - `src/http/retry.ts` implements retry/backoff behavior, especially for **429 Too Many Requests** responses.
   - This layer is the boundary between scraper logic and the external service.

5. **JSF/PrimeFaces compatibility layer**
   - `src/jsf/viewState.ts` extracts and tracks the rotating **ViewState** token required by JSF.
   - `src/jsf/partialResponse.ts` unwraps JSF partial responses, which are not plain HTML but XML envelopes containing the actual content.
   - `src/jsf/formParams.ts` builds the POST bodies needed for search, pagination, export, and file download actions.
   - These modules isolate all site-specific protocol quirks.

6. **Scraping and extraction layer**
   - `src/scraper/searchClient.ts` handles the search and pagination lifecycle, including moving through pages by offset.
   - `src/scraper/recordParser.ts` converts table rows into structured record objects.
   - `src/scraper/excelDownloader.ts` triggers and saves the site’s full Excel export.
   - `src/scraper/pdfDownloader.ts` downloads each document PDF and validates the result before persisting it.

7. **Persistence layer**
   - `src/persistence/recordStore.ts` writes extracted records to JSON output.
   - `src/persistence/failureLog.ts` records documents that could not be downloaded so they can be retried later.
   - `src/persistence/fileNaming.ts` centralizes output naming, especially for PDFs and exported files, so filenames remain stable and deduplicated.
   - These modules define the filesystem boundary of the application.

8. **Shared configuration and types**
   - `src/config.ts` holds runtime configuration such as endpoints, delays, output locations, and other scraper parameters.
   - `src/types.ts` contains shared data models used across HTTP, parsing, scraping, and persistence.
   - This keeps the rest of the code strongly typed and avoids scattering constants or shape definitions.

9. **Utility support**
   - `src/utils/logger.ts` provides structured logging for progress, warnings, and errors.
   - `src/utils/fs.ts` handles directory creation and filesystem setup.
   - `src/utils/sleep.ts` supports backoff and pacing between requests.
   - These utilities are small, generic support services used across the pipeline.

10. **Retry mode as a separate recovery path**
    - The repo’s retry flow re-establishes a fresh JSF session, navigates directly to the page offsets recorded in `failures.json`, and retries only the failed PDFs for those pages.
    - This is important architecturally: retry is not just “repeat the same request,” but a stateful recovery path that must restore the correct page/view state before replaying downloads.
    - The result is a two-phase system: **primary crawl** plus **targeted recovery**.

11. **External dependency and protocol constraints**
    - The only real external system is the OEFA repository site itself, which dictates the architecture because of JSF statefulness, partial responses, and request throttling.
    - The code structure is shaped around those constraints: session management, ViewState rotation, paginated search, export endpoints, and per-record download actions.

12. **Repository shape and maintainability**
    - The project is intentionally layered: transport, JSF protocol handling, scraping orchestration, persistence, and utilities.
    - That separation makes the architecture diagram straightforward: a single orchestrator in the middle, with well-defined dependencies outward to HTTP/JSF helpers and filesystem persistence.
    - The repo also includes documentation (`docs/SOLUTION_DESIGN.md`, `docs/SITE_NOTES.md`) that records the site contract and design decisions, reinforcing that most complexity is in protocol adaptation rather than general crawling logic.

Main entry points: [`src/scraper/pipeline.ts`](https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/pipeline.ts) (Pipeline), [`src/http/client.ts`](https://github.com/vzepec/scraper-challenge/blob/master/src/http/client.ts) (HTTP client), [`src/scraper/searchClient.ts`](https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/searchClient.ts) (Search client), [`src/jsf/viewState.ts`](https://github.com/vzepec/scraper-challenge/blob/master/src/jsf/viewState.ts) (ViewState)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Entry & orchestration"]
  node_n1["index.ts<br/>entrypoint<br/>[index.ts]"]
  node_n2["Pipeline<br/>orchestrator<br/>[pipeline.ts]"]
end

subgraph group_g2["HTTP & JSF protocol"]
  node_n3["HTTP client<br/>transport<br/>[client.ts]"]
  node_n4["Rate limiter<br/>throttle<br/>[rateLimiter.ts]"]
  node_n5["Retry logic<br/>backoff<br/>[retry.ts]"]
  node_n6["ViewState<br/>jsf token<br/>[viewState.ts]"]
  node_n7["Partial response<br/>jsf parser<br/>[partialResponse.ts]"]
  node_n8["Form params<br/>jsf builder<br/>[formParams.ts]"]
end

subgraph group_g3["Scraping stages"]
  node_n9["Search client<br/>crawl flow<br/>[searchClient.ts]"]
  node_n10["Record parser<br/>row parser<br/>[recordParser.ts]"]
  node_n11["Excel export<br/>exporter<br/>[excelDownloader.ts]"]
  node_n12["PDF downloader<br/>document fetch<br/>[pdfDownloader.ts]"]
end

subgraph group_g4["Persistence"]
  node_n13["Record store<br/>json output<br/>[recordStore.ts]"]
  node_n14["Failure log<br/>retry state<br/>[failureLog.ts]"]
  node_n15["File naming<br/>[fileNaming.ts]"]
end

subgraph group_g5["Shared support"]
  node_n16["Config<br/>[config.ts]"]
  node_n17["Types<br/>shared models<br/>[types.ts]"]
  node_n18["Logger<br/>logging<br/>[logger.ts]"]
  node_n19["Filesystem<br/>fs helpers<br/>[fs.ts]"]
  node_n20["Sleep<br/>timing<br/>[sleep.ts]"]
end

node_n1 -->|"starts"| node_n2
node_n2 -->|"reads config"| node_n16
node_n2 -->|"logs"| node_n18
node_n2 -->|"prepares output"| node_n19
node_n2 -->|"requests"| node_n3
node_n2 -->|"drives crawl"| node_n9
node_n2 -->|"exports"| node_n11
node_n2 -->|"downloads"| node_n12
node_n2 -->|"persists records"| node_n13
node_n2 -->|"records failures"| node_n14
node_n2 -->|"names files"| node_n15
node_n2 -->|"paces work"| node_n20
node_n3 -->|"throttled by"| node_n4
node_n3 -->|"retried by"| node_n5
node_n3 -->|"tracks"| node_n6
node_n3 -->|"unwraps"| node_n7
node_n3 -->|"uses config"| node_n16
node_n9 -->|"updates viewstate"| node_n6
node_n9 -->|"builds actions"| node_n8
node_n9 -->|"parses rows"| node_n10
node_n11 -->|"builds export"| node_n8
node_n12 -->|"builds download"| node_n8
node_n12 -->|"names outputs"| node_n15
node_n13 -->|"serializes models"| node_n17
node_n14 -->|"stores state"| node_n17
node_n15 -->|"writes paths"| node_n19
node_n6 -.->|"interprets"| node_n7
node_n8 -->|"embeds token"| node_n6
node_n16 -.->|"defines shapes"| node_n17

click node_n1 "https://github.com/vzepec/scraper-challenge/blob/master/src/index.ts"
click node_n2 "https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/pipeline.ts"
click node_n3 "https://github.com/vzepec/scraper-challenge/blob/master/src/http/client.ts"
click node_n4 "https://github.com/vzepec/scraper-challenge/blob/master/src/http/rateLimiter.ts"
click node_n5 "https://github.com/vzepec/scraper-challenge/blob/master/src/http/retry.ts"
click node_n6 "https://github.com/vzepec/scraper-challenge/blob/master/src/jsf/viewState.ts"
click node_n7 "https://github.com/vzepec/scraper-challenge/blob/master/src/jsf/partialResponse.ts"
click node_n8 "https://github.com/vzepec/scraper-challenge/blob/master/src/jsf/formParams.ts"
click node_n9 "https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/searchClient.ts"
click node_n10 "https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/recordParser.ts"
click node_n11 "https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/excelDownloader.ts"
click node_n12 "https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/pdfDownloader.ts"
click node_n13 "https://github.com/vzepec/scraper-challenge/blob/master/src/persistence/recordStore.ts"
click node_n14 "https://github.com/vzepec/scraper-challenge/blob/master/src/persistence/failureLog.ts"
click node_n15 "https://github.com/vzepec/scraper-challenge/blob/master/src/persistence/fileNaming.ts"
click node_n16 "https://github.com/vzepec/scraper-challenge/blob/master/src/config.ts"
click node_n17 "https://github.com/vzepec/scraper-challenge/blob/master/src/types.ts"
click node_n18 "https://github.com/vzepec/scraper-challenge/blob/master/src/utils/logger.ts"
click node_n19 "https://github.com/vzepec/scraper-challenge/blob/master/src/utils/fs.ts"
click node_n20 "https://github.com/vzepec/scraper-challenge/blob/master/src/utils/sleep.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2 toneBlue
class node_n3,node_n4,node_n5,node_n6,node_n7,node_n8 toneAmber
class node_n9,node_n10,node_n11,node_n12 toneMint
class node_n13,node_n14,node_n15 toneRose
class node_n16,node_n17,node_n18,node_n19,node_n20 toneIndigo
```

## Components

### Entry & orchestration

Program entrypoint and the main crawl/retry pipeline.

- [index.ts](https://github.com/vzepec/scraper-challenge/blob/master/src/index.ts): entrypoint · CLI/program entry that starts the scraper. · `src/index.ts`
- [Pipeline](https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/pipeline.ts): orchestrator · Coordinates session bootstrap, search, pagination, parsing, exports, downloads, persistence, and retries. · `src/scraper/pipeline.ts`

### HTTP & JSF protocol

Session-aware transport plus JSF/PrimeFaces compatibility helpers.

- [HTTP client](https://github.com/vzepec/scraper-challenge/blob/master/src/http/client.ts): transport · Session-aware HTTP requests to the OEFA site. · `src/http/client.ts`
- [Rate limiter](https://github.com/vzepec/scraper-challenge/blob/master/src/http/rateLimiter.ts): throttle · Controls pacing and concurrency to avoid overwhelming the site. · `src/http/rateLimiter.ts`
- [Retry logic](https://github.com/vzepec/scraper-challenge/blob/master/src/http/retry.ts): backoff · Retries transient HTTP failures, especially 429 responses. · `src/http/retry.ts`
- [ViewState](https://github.com/vzepec/scraper-challenge/blob/master/src/jsf/viewState.ts): jsf token · Tracks the rotating JSF ViewState value. · `src/jsf/viewState.ts`
- [Partial response](https://github.com/vzepec/scraper-challenge/blob/master/src/jsf/partialResponse.ts): jsf parser · Unwraps JSF XML partial responses into usable content. · `src/jsf/partialResponse.ts`
- [Form params](https://github.com/vzepec/scraper-challenge/blob/master/src/jsf/formParams.ts): jsf builder · Builds POST bodies for search, paging, export, and downloads. · `src/jsf/formParams.ts`

### Scraping stages

Search, parsing, export, and document download steps.

- [Search client](https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/searchClient.ts): crawl flow · Runs search and paginates through the result set. · `src/scraper/searchClient.ts`
- [Record parser](https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/recordParser.ts): row parser · Converts table rows into structured records. · `src/scraper/recordParser.ts`
- [Excel export](https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/excelDownloader.ts): exporter · Triggers and saves the full Excel export. · `src/scraper/excelDownloader.ts`
- [PDF downloader](https://github.com/vzepec/scraper-challenge/blob/master/src/scraper/pdfDownloader.ts): document fetch · Downloads and validates per-record PDFs. · `src/scraper/pdfDownloader.ts`

### Persistence

Filesystem outputs and retry state.

- [Record store](https://github.com/vzepec/scraper-challenge/blob/master/src/persistence/recordStore.ts): json output · Writes extracted records to JSON files. · `src/persistence/recordStore.ts`
- [Failure log](https://github.com/vzepec/scraper-challenge/blob/master/src/persistence/failureLog.ts): retry state · Stores PDF download failures for later replay. · `src/persistence/failureLog.ts`
- [File naming](https://github.com/vzepec/scraper-challenge/blob/master/src/persistence/fileNaming.ts): naming · Centralizes stable names for PDFs and exports. · `src/persistence/fileNaming.ts`

### Shared support

Configuration, types, and utilities used across the scraper.

- [Config](https://github.com/vzepec/scraper-challenge/blob/master/src/config.ts): runtime config · Holds endpoints, delays, paths, and scraper parameters. · `src/config.ts`
- [Types](https://github.com/vzepec/scraper-challenge/blob/master/src/types.ts): shared models · Shared data models used across the application. · `src/types.ts`
- [Logger](https://github.com/vzepec/scraper-challenge/blob/master/src/utils/logger.ts): logging · Structured logging for progress, warnings, and errors. · `src/utils/logger.ts`
- [Filesystem](https://github.com/vzepec/scraper-challenge/blob/master/src/utils/fs.ts): fs helpers · Directory creation and setup helpers. · `src/utils/fs.ts`
- [Sleep](https://github.com/vzepec/scraper-challenge/blob/master/src/utils/sleep.ts): timing · Pacing and backoff delay helper. · `src/utils/sleep.ts`

## Connections

- index.ts → Pipeline: starts · Entrypoint launches the main pipeline.
- Pipeline → Config: reads config · Pipeline uses runtime settings for endpoints, pacing, and output paths.
- Pipeline → Logger: logs · Pipeline reports progress and failures.
- Pipeline → Filesystem: prepares output · Pipeline ensures output directories exist.
- Pipeline → HTTP client: requests · Pipeline performs session-aware HTTP calls through the client.
- Pipeline → Search client: drives crawl · Pipeline delegates search and pagination flow.
- Pipeline → Excel export: exports · Pipeline triggers the Excel export phase.
- Pipeline → PDF downloader: downloads · Pipeline initiates per-record PDF fetches.
- Pipeline → Record store: persists records · Pipeline writes extracted records to disk.
- Pipeline → Failure log: records failures · Pipeline stores failed PDF downloads for retry.
- Pipeline → File naming: names files · Pipeline relies on stable output naming.
- Pipeline → Sleep: paces work · Pipeline uses sleep for delays and backoff timing.
- HTTP client → Rate limiter: throttled by · HTTP client is constrained by rate limiting.
- HTTP client → Retry logic: retried by · HTTP client operations use retry/backoff behavior.
- HTTP client → ViewState: tracks · HTTP client works with rotating JSF ViewState.
- HTTP client → Partial response: unwraps · HTTP responses may need partial-response extraction.
- HTTP client → Config: uses config · Client needs endpoints and timeouts.
- Search client → ViewState: updates viewstate · Search and paging must keep the JSF token current.
- Search client → Form params: builds actions · Search client creates request bodies for search and paging.
- Search client → Record parser: parses rows · Search pages are converted into records.
- Excel export → Form params: builds export · Excel export uses JSF form parameters.
- PDF downloader → Form params: builds download · PDF fetches use JSF action parameters.
- PDF downloader → File naming: names outputs · Downloader uses naming helpers for saved PDFs.
- Record store → Types: serializes models · Record persistence writes shared record shapes.
- Failure log → Types: stores state · Failure log persists failed download metadata.
- File naming → Filesystem: writes paths · Naming helpers are used when creating filesystem outputs.
- ViewState → Partial response: interprets · ViewState handling depends on partial-response extraction.
- Form params → ViewState: embeds token · Form bodies include the current ViewState.
- Config → Types: defines shapes · Config and types establish shared runtime contracts.