How lugonesnicolas/ndevscrap works
NDevScrap is a modular retail-data acquisition CLI. Its main workflow runs the DIA store definition, fetches the VTEX catalog and optionally authorized ClubDIA coupons, normalizes and validates component records, then publishes component-aware snapshots and run evidence. The graph also shows the internal contracts and transport boundary. README capabilities guide the broader map; sampled excerpts do not expose every CLI, runner, or observability call, so uncertain wiring is omitted.
Main entry points: runner.py Run orchestrator · clubdia.py ClubDIA connector · cli.py CLI · vtex.py VTEX connector
Components 21
Entry and composition
- CLI
src/ndevscrap/cli.py - Configuration
src/ndevscrap/config.py - Store registry
src/ndevscrap/stores/__init__.py - Typed contracts
src/ndevscrap/contracts.py
Store acquisition
- DIA composition
src/ndevscrap/stores/dia.py - VTEX connector
src/ndevscrap/connectors/vtex.py - ClubDIA connector
src/ndevscrap/connectors/clubdia.py - Session provider
src/ndevscrap/session.py - HAR extraction
src/ndevscrap/stores/dia_session.py - HTTP transport
src/ndevscrap/transport.py
Run processing
- Run orchestrator
src/ndevscrap/runner.py - Quality controls
src/ndevscrap/quality.py - Snapshot models
src/ndevscrap/models.py
Snapshots and evidence
- Snapshot storage
src/ndevscrap/storage.py - Run observability
src/ndevscrap/observability.py - Published snapshots
- Run manifest
Other components
- Operator
- VTEX APIs
- ClubDIA API
- Authorized session file
Connections 26
- Operator → CLI: invokes
- CLI → Configuration: loads settings
- CLI → Store registry: selects store
- CLI → Run orchestrator: starts run
- Store registry → DIA composition: creates definition
- Run orchestrator → DIA composition: requests components
- DIA composition → VTEX connector: composes catalog
- DIA composition → ClubDIA connector: composes optional feed
- Run orchestrator → Typed contracts: uses run contracts
- VTEX connector → HTTP transport: sends requests
- ClubDIA connector → HTTP transport: sends requests
- VTEX connector → VTEX APIs: fetches catalog
- ClubDIA connector → ClubDIA API: fetches coupons
- ClubDIA connector → Session provider: loads credentials
- Session provider → Authorized session file: reads
- Operator → HAR extraction: provides authorized HAR
- HAR extraction → Authorized session file: extracts session
- Run orchestrator → HTTP transport: builds transport
- Run orchestrator → Snapshot storage: writes and publishes
- Run orchestrator → Quality controls: checks records
- VTEX connector → Snapshot models: normalizes products
- ClubDIA connector → Snapshot models: normalizes coupons
- Snapshot storage → Published snapshots: publishes components
- Snapshot storage → Run manifest: records run outcome
- Run orchestrator → Run observability: reports run
- Run observability → Run manifest: records metrics