# amitchakraborty123/tecno-agricola-scraper architecture

> Architecture diagram of the GitHub repository amitchakraborty123/tecno-agricola-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amitchakraborty123/tecno-agricola-scraper
- Repository: https://github.com/amitchakraborty123/tecno-agricola-scraper

## Overview

1. **Project type and scope**
   - This is a small, single-purpose Python scraping tool, not a multi-service application.
   - The only external system in the architecture is the target website, `tecnoagricolamodena.it`, from which product catalog data is collected and exported.

2. **Primary execution entry point**
   - `src/scraper.py` is the runtime entry point.
   - The README shows it is run interactively from the `src/` directory and prompts the user to choose between:
     - scraping all categories, or
     - scraping one category via a provided URL.

3. **Core processing pipeline**
   - The app follows a straightforward flow:
     1. read the user’s scraping mode,
     2. determine the category URL set,
     3. request category and product pages,
     4. parse product data,
     5. aggregate results,
     6. export a CSV to `output/products.csv`.
   - The output is tabular product data with fields like Link, Name, Price, Image, and Availability.

4. **Scraping and parsing layer**
   - The scraper uses Python HTTP and HTML parsing libraries listed in the README: `requests`, `beautifulsoup4`, and `lxml`.
   - This layer is responsible for downloading pages, traversing pagination, and extracting product cards/details from HTML.

5. **Data transformation and export**
   - Product records are normalized into a structured dataset before export.
   - `pandas` is used for the final tabular representation and CSV generation, which is the main persistence artifact of the repo.
   - The repository also includes `sample_output/Products_Data.csv` as a reference output.

6. **Configuration boundary**
   - `src/config.py` centralizes runtime settings and constants.
   - The presence of `.env.example` suggests some settings can be externalized, even if the project is primarily file-and-CLI driven.
   - This creates a clean boundary between scraping logic and environment-specific values such as paths or site-specific constants.

7. **Utility/shared helpers**
   - `src/utils.py` likely contains shared support functions used by the scraper, such as formatting, URL handling, logging helpers, or file/output helpers.
   - In architecture terms, this is the “support” layer beneath the main scraper workflow.

8. **Reliability and operational behavior**
   - The README explicitly calls out session reuse, automatic retries, and logging.
   - That means the scraper is designed to be resilient to transient network failures and to reuse HTTP state efficiently during long catalog crawls.
   - The `logs/` directory exists as a runtime artifact boundary for operational diagnostics.

9. **Testing strategy**
   - `tests/test_scraper.py` validates scraper behavior using fixture HTML in `tests/fixtures/sample_product.html`.
   - This indicates the tests are parser-centric and likely avoid live network dependency by feeding saved HTML into the scraping logic.
   - The architecture therefore separates network access from HTML extraction enough to make unit testing feasible.

10. **Repository output and artifact boundaries**
   - `output/` is the active destination for generated CSVs.
   - `sample_output/` holds a reference dataset for expected output shape.
   - `screenshots/` documents the workflow visually but does not affect runtime behavior.

11. **Technology stack and runtime**
   - The project targets Python 3.10+ and is packaged with `pyproject.toml` plus `requirements.txt`.
   - Tooling includes `pytest` for tests and `ruff`/`black` for quality and formatting, which matters mostly for maintainability rather than runtime architecture.

12. **Diagram framing**
   - If you draw the architecture, the main boxes are:
     - User/CLI input
     - `scraper.py` orchestration
     - HTTP fetching to the external ecommerce site
     - HTML parsing/extraction
     - Data assembly/export via `pandas`
     - CSV output under `output/`
     - tests using local HTML fixtures
   - The repo is essentially a linear ETL-style scraper with a thin configuration layer and a small test harness.

Main entry points: [`src/scraper.py`](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py) (Scraper), [`tests/test_scraper.py`](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/tests/test_scraper.py) (Scraper tests), [`src/config.py`](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/config.py) (Config), [`src/utils.py`](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/utils.py) (Utils)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_runtime["Runtime"]
  node_node_user(("User<br/>cli actor"))
  node_node_scraper["Scraper<br/>entry orchestration<br/>[scraper.py]"]
  node_node_mode{{"Mode select<br/>input routing<br/>[scraper.py]"}}
  node_node_catalog["Catalog flow<br/>crawl workflow<br/>[scraper.py]"]
  node_node_fetch["HTTP fetch<br/>network client<br/>[scraper.py]"]
  node_node_parse["HTML parse<br/>extraction<br/>[scraper.py]"]
  node_node_aggregate["Aggregate<br/>record assembly<br/>[scraper.py]"]
  node_node_export["CSV export<br/>tabular output<br/>[scraper.py]"]
end

subgraph group_grp_support["Support"]
  node_node_config["Config<br/>settings module<br/>[config.py]"]
  node_node_utils["Utils<br/>shared helpers<br/>[utils.py]"]
end

subgraph group_grp_testing["Testing"]
  node_node_tests["Scraper tests<br/>pytest suite<br/>[test_scraper.py]"]
  node_node_fixture["HTML fixture<br/>sample page"]
end

subgraph group_grp_artifacts["Artifacts"]
  node_node_output["Output CSV<br/>generated artifact<br/>[.gitkeep]"]
  node_node_sample["Sample CSV<br/>reference artifact<br/>[Products_Data.csv]"]
  node_node_logs["Logs<br/>runtime artifact<br/>[.gitkeep]"]
end

subgraph group_grp_external["External"]
  node_node_site(("Tecno Agricola<br/>target website"))
end

node_node_user -->|"choose"| node_node_mode
node_node_mode -->|"read settings"| node_node_config
node_node_scraper -->|"prompt"| node_node_mode
node_node_scraper -->|"orchestrate"| node_node_catalog
node_node_catalog -->|"request pages"| node_node_fetch
node_node_fetch -->|"GET"| node_node_site
node_node_catalog -->|"parse pages"| node_node_parse
node_node_parse -->|"extract from"| node_node_site
node_node_parse -->|"records"| node_node_aggregate
node_node_aggregate -->|"helpers"| node_node_utils
node_node_aggregate -->|"tabularize"| node_node_export
node_node_export -->|"write"| node_node_output
node_node_export -.->|"shape"| node_node_sample
node_node_scraper -->|"log"| node_node_logs
node_node_tests -->|"uses"| node_node_fixture
node_node_tests -->|"verify"| node_node_parse
node_node_tests -.->|"exercise"| node_node_scraper

click node_node_scraper "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py"
click node_node_mode "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py"
click node_node_catalog "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py"
click node_node_fetch "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py"
click node_node_parse "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py"
click node_node_aggregate "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py"
click node_node_export "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py"
click node_node_config "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/config.py"
click node_node_utils "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/utils.py"
click node_node_tests "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/tests/test_scraper.py"
click node_node_fixture "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/tests/fixtures/sample_product.html"
click node_node_output "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/output/.gitkeep"
click node_node_sample "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/sample_output/Products_Data.csv"
click node_node_logs "https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/logs/.gitkeep"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_user,node_node_scraper,node_node_mode,node_node_catalog,node_node_fetch,node_node_parse,node_node_aggregate,node_node_export toneBlue
class node_node_config,node_node_utils toneAmber
class node_node_tests,node_node_fixture toneMint
class node_node_output,node_node_sample,node_node_logs toneRose
class node_node_site toneIndigo
```

## Components

### Runtime

Interactive scraping workflow and core code

- **User**: cli actor · Chooses scrape mode and provides category URL when needed
- [Scraper](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py): entry orchestration · Main runtime entry point and crawl controller · `src/scraper.py`
- [Mode select](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py): input routing · Determines whether to scrape all categories or one category · `src/scraper.py`
- [Catalog flow](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py): crawl workflow · Builds category URL set and drives pagination across product listings · `src/scraper.py`
- [HTTP fetch](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py): network client · Downloads category and product pages with session reuse and retries · `src/scraper.py`
- [HTML parse](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py): extraction · Extracts product cards and product details from page HTML · `src/scraper.py`
- [Aggregate](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py): record assembly · Normalizes scraped fields into a structured product dataset · `src/scraper.py`
- [CSV export](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/scraper.py): tabular output · Converts records to a dataframe and writes the final CSV · `src/scraper.py`

### Support

Configuration and shared helpers

- [Config](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/config.py): settings module · Centralizes runtime constants and site-specific settings · `src/config.py`
- [Utils](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/src/utils.py): shared helpers · Common support functions for formatting, URLs, logging, or output handling · `src/utils.py`

### Testing

Unit tests and HTML fixtures

- [Scraper tests](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/tests/test_scraper.py): pytest suite · Validates scraper behavior against local fixture HTML · `tests/test_scraper.py`
- [HTML fixture](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/tests/fixtures/sample_product.html): sample page · Saved product page HTML used for parser-centric testing · `tests/fixtures/sample_product.html`

### Artifacts

Generated and reference outputs

- [Output CSV](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/output/.gitkeep): generated artifact · Primary generated product dataset at output/products.csv · `output/.gitkeep`
- [Sample CSV](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/sample_output/Products_Data.csv): reference artifact · Reference output dataset showing expected tabular shape · `sample_output/Products_Data.csv`
- [Logs](https://github.com/amitchakraborty123/tecno-agricola-scraper/blob/master/logs/.gitkeep): runtime artifact · Operational diagnostics generated during scraping runs · `logs/.gitkeep`

### External

Target website being scraped

- **Tecno Agricola**: target website · External ecommerce site whose catalog pages are scraped

## Connections

- User → Mode select: choose · User selects scraping mode and may supply a category URL
- Mode select → Config: read settings · Mode handling depends on centralized constants and paths
- Scraper → Mode select: prompt · Entry point collects interactive input
- Scraper → Catalog flow: orchestrate · Main script coordinates the crawl workflow
- Catalog flow → HTTP fetch: request pages · Catalog flow fetches listing and product pages
- HTTP fetch → Tecno Agricola: GET · HTTP requests target the ecommerce site
- Catalog flow → HTML parse: parse pages · Fetched HTML is handed to extraction logic
- HTML parse → Tecno Agricola: extract from · Parsed content originates from site HTML
- HTML parse → Aggregate: records · Extraction yields structured product fields
- Aggregate → Utils: helpers · Normalization may rely on shared helper functions
- Aggregate → CSV export: tabularize · Aggregated records are prepared for dataframe output
- CSV export → Output CSV: write · Final CSV is written to the output directory
- CSV export → Sample CSV: shape · Reference CSV reflects the expected exported schema
- Scraper → Logs: log · Operational logging is emitted during crawl runs
- Scraper tests → HTML fixture: uses · Tests load saved HTML instead of live pages
- Scraper tests → HTML parse: verify · Tests focus on parser behavior and extraction results
- Scraper tests → Scraper: exercise · Test suite covers scraper behavior through the runtime code path