amitchakraborty123/tecno-agricola-scraper

1. Project type and scope

- This is a small, single-purpose Python scraping tool, not a multi-service application.

- The only external system in the architecture is the target website, tecnoagricolamodena.it, from which product catalog data is collected and exported.

2. Primary execution entry point

- src/scraper.py is the runtime entry point.

- The README shows it is run interactively from the src/ directory and prompts the user to choose between:

- scraping all categories, or

- scraping one category via a provided URL.

3. Core processing pipeline

- The app follows a straightforward flow:

1. read the user’s scraping mode,

2. determine the category URL set,

3. request category and product pages,

4. parse product data,

5. aggregate results,

6. export a CSV to output/products.csv.

- The output is tabular product data with fields like Link, Name, Price, Image, and Availability.

4. Scraping and parsing layer

- The scraper uses Python HTTP and HTML parsing libraries listed in the README: requests, beautifulsoup4, and lxml.

- This layer is responsible for downloading pages, traversing pagination, and extracting product cards/details from HTML.

5. Data transformation and export

- Product records are normalized into a structured dataset before export.

- pandas is used for the final tabular representation and CSV generation, which is the main persistence artifact of the repo.

- The repository also includes sample_output/Products_Data.csv as a reference output.

6. Configuration boundary

- src/config.py centralizes runtime settings and constants.

- The presence of .env.example suggests some settings can be externalized, even if the project is primarily file-and-CLI driven.

- This creates a clean boundary between scraping logic and environment-specific values such as paths or site-specific constants.

7. Utility/shared helpers

- src/utils.py likely contains shared support functions used by the scraper, such as formatting, URL handling, logging helpers, or file/output helpers.

- In architecture terms, this is the “support” layer beneath the main scraper workflow.

8. Reliability and operational behavior

- The README explicitly calls out session reuse, automatic retries, and logging.

- That means the scraper is designed to be resilient to transient network failures and to reuse HTTP state efficiently during long catalog crawls.

- The logs/ directory exists as a runtime artifact boundary for operational diagnostics.

9. Testing strategy

- tests/test_scraper.py validates scraper behavior using fixture HTML in tests/fixtures/sample_product.html.

- This indicates the tests are parser-centric and likely avoid live network dependency by feeding saved HTML into the scraping logic.

- The architecture therefore separates network access from HTML extraction enough to make unit testing feasible.

10. Repository output and artifact boundaries

- output/ is the active destination for generated CSVs.

- sample_output/ holds a reference dataset for expected output shape.

- screenshots/ documents the workflow visually but does not affect runtime behavior.

11. Technology stack and runtime

- The project targets Python 3.10+ and is packaged with pyproject.toml plus requirements.txt.

- Tooling includes pytest for tests and ruff/black for quality and formatting, which matters mostly for maintainability rather than runtime architecture.

12. Diagram framing

- If you draw the architecture, the main boxes are:

- User/CLI input

- scraper.py orchestration

- HTTP fetching to the external ecommerce site

- HTML parsing/extraction

- Data assembly/export via pandas

- CSV output under output/

- tests using local HTML fixtures

- The repo is essentially a linear ETL-style scraper with a thin configuration layer and a small test harness.