# lugonesnicolas/ndevscrap architecture

> Architecture diagram of the GitHub repository lugonesnicolas/ndevscrap, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lugonesnicolas/ndevscrap
- Repository: https://github.com/lugonesnicolas/ndevscrap

## Overview

NDevScrap is a modular retail-data acquisition CLI. Its main workflow runs the DIA store definition, fetches the VTEX catalog and optionally authorized ClubDIA coupons, normalizes and validates component records, then publishes component-aware snapshots and run evidence. The graph also shows the internal contracts and transport boundary. README capabilities guide the broader map; sampled excerpts do not expose every CLI, runner, or observability call, so uncertain wiring is omitted.

Main entry points: [`src/ndevscrap/runner.py`](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/runner.py) (Run orchestrator), [`src/ndevscrap/connectors/clubdia.py`](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/connectors/clubdia.py) (ClubDIA connector), [`src/ndevscrap/cli.py`](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/cli.py) (CLI), [`src/ndevscrap/connectors/vtex.py`](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/connectors/vtex.py) (VTEX connector)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and composition"]
  node_cli["CLI<br/>[cli.py]"]
  node_config["Configuration<br/>[config.py]"]
  node_registry["Store registry<br/>[__init__.py]"]
  node_contracts["Typed contracts<br/>[contracts.py]"]
end

subgraph group_acquisition["Store acquisition"]
  node_dia["DIA composition<br/>[dia.py]"]
  node_vtex["VTEX connector<br/>[vtex.py]"]
  node_clubdia["ClubDIA connector<br/>[clubdia.py]"]
  node_session["Session provider<br/>[session.py]"]
  node_har["HAR extraction<br/>[dia_session.py]"]
  node_transport["HTTP transport<br/>[transport.py]"]
end

subgraph group_processing["Run processing"]
  node_runner["Run orchestrator<br/>[runner.py]"]
  node_quality["Quality controls<br/>[quality.py]"]
  node_models["Snapshot models<br/>[models.py]"]
end

subgraph group_persistence["Snapshots and evidence"]
  node_storage[("Snapshot storage<br/>[storage.py]")]
  node_observability["Run observability<br/>[observability.py]"]
  node_snapshot["Published snapshots"]
  node_manifest["Run manifest"]
end

node_operator(("Operator"))
node_vtex_api{{"VTEX APIs"}}
node_clubdia_api{{"ClubDIA API"}}
node_session_file["Authorized session file"]

node_operator -->|"invokes"| node_cli
node_cli -->|"loads settings"| node_config
node_cli -->|"selects store"| node_registry
node_cli -->|"starts run"| node_runner
node_registry -->|"creates definition"| node_dia
node_runner -->|"requests components"| node_dia
node_dia -->|"composes catalog"| node_vtex
node_dia -.->|"composes optional feed"| node_clubdia
node_runner -->|"uses run contracts"| node_contracts
node_vtex -->|"sends requests"| node_transport
node_clubdia -->|"sends requests"| node_transport
node_vtex -->|"fetches catalog"| node_vtex_api
node_clubdia -.->|"fetches coupons"| node_clubdia_api
node_clubdia -->|"loads credentials"| node_session
node_session -->|"reads"| node_session_file
node_operator -.->|"provides authorized HAR"| node_har
node_har -.->|"extracts session"| node_session_file
node_runner -->|"builds transport"| node_transport
node_runner -->|"writes and publishes"| node_storage
node_runner -->|"checks records"| node_quality
node_vtex -->|"normalizes products"| node_models
node_clubdia -->|"normalizes coupons"| node_models
node_storage -->|"publishes components"| node_snapshot
node_storage -->|"records run outcome"| node_manifest
node_runner -->|"reports run"| node_observability
node_observability -->|"records metrics"| node_manifest

click node_cli "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/cli.py"
click node_config "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/config.py"
click node_registry "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/stores/__init__.py"
click node_contracts "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/contracts.py"
click node_runner "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/runner.py"
click node_dia "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/stores/dia.py"
click node_vtex "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/connectors/vtex.py"
click node_clubdia "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/connectors/clubdia.py"
click node_session "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/session.py"
click node_har "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/stores/dia_session.py"
click node_transport "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/transport.py"
click node_quality "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/quality.py"
click node_models "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/models.py"
click node_storage "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/storage.py"
click node_observability "https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/observability.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_registry,node_contracts toneBlue
class node_dia,node_vtex,node_clubdia,node_session,node_har,node_transport toneAmber
class node_runner,node_quality,node_models,node_vtex_api,node_clubdia_api toneMint
class node_storage,node_observability,node_snapshot,node_manifest toneRose
class node_operator,node_session_file toneIndigo
```

## Components

### Entry and composition

- [CLI](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/cli.py): `src/ndevscrap/cli.py`
- [Configuration](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/config.py): `src/ndevscrap/config.py`
- [Store registry](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/stores/__init__.py): `src/ndevscrap/stores/__init__.py`
- [Typed contracts](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/contracts.py): `src/ndevscrap/contracts.py`

### Store acquisition

- [DIA composition](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/stores/dia.py): `src/ndevscrap/stores/dia.py`
- [VTEX connector](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/connectors/vtex.py): `src/ndevscrap/connectors/vtex.py`
- [ClubDIA connector](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/connectors/clubdia.py): `src/ndevscrap/connectors/clubdia.py`
- [Session provider](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/session.py): `src/ndevscrap/session.py`
- [HAR extraction](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/stores/dia_session.py): `src/ndevscrap/stores/dia_session.py`
- [HTTP transport](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/transport.py): `src/ndevscrap/transport.py`

### Run processing

- [Run orchestrator](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/runner.py): `src/ndevscrap/runner.py`
- [Quality controls](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/quality.py): `src/ndevscrap/quality.py`
- [Snapshot models](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/models.py): `src/ndevscrap/models.py`

### Snapshots and evidence

- [Snapshot storage](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/storage.py): `src/ndevscrap/storage.py`
- [Run observability](https://github.com/lugonesnicolas/ndevscrap/blob/main/src/ndevscrap/observability.py): `src/ndevscrap/observability.py`
- **Published snapshots**
- **Run manifest**

### Other components

- **Operator**
- **VTEX APIs**
- **ClubDIA API**
- **Authorized session file**

## Connections

- Operator → CLI: invokes
- CLI → Configuration: loads settings
- CLI → Store registry: selects store
- CLI → Run orchestrator: starts run
- Store registry → DIA composition: creates definition
- Run orchestrator → DIA composition: requests components
- DIA composition → VTEX connector: composes catalog
- DIA composition → ClubDIA connector: composes optional feed
- Run orchestrator → Typed contracts: uses run contracts
- VTEX connector → HTTP transport: sends requests
- ClubDIA connector → HTTP transport: sends requests
- VTEX connector → VTEX APIs: fetches catalog
- ClubDIA connector → ClubDIA API: fetches coupons
- ClubDIA connector → Session provider: loads credentials
- Session provider → Authorized session file: reads
- Operator → HAR extraction: provides authorized HAR
- HAR extraction → Authorized session file: extracts session
- Run orchestrator → HTTP transport: builds transport
- Run orchestrator → Snapshot storage: writes and publishes
- Run orchestrator → Quality controls: checks records
- VTEX connector → Snapshot models: normalizes products
- ClubDIA connector → Snapshot models: normalizes coupons
- Snapshot storage → Published snapshots: publishes components
- Snapshot storage → Run manifest: records run outcome
- Run orchestrator → Run observability: reports run
- Run observability → Run manifest: records metrics