# friendlyuser/flipp_flyer_parser architecture

> Architecture diagram of the GitHub repository friendlyuser/flipp_flyer_parser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/friendlyuser/flipp_flyer_parser
- Repository: https://github.com/friendlyuser/flipp_flyer_parser

## Overview

##### Main Components
- Entry Points & Controllers: Files like main.py, run_batch.py, and legacy_code.py serve as the primary execution scripts. They initialize the web scraping tasks, manage batch operations, and maintain backward–compatible functionality (legacy_code.py).
- Data Processing Modules: There are specialized scripts such as analyze_flyers.py, convert.py, and purge.py. These modules handle tasks like analyzing flyer content, converting raw scraped data into a structured format, and purging duplicate or outdated records.
- Utilities: The utils folder (containing ref.csv) provides auxiliary data and perhaps helper functions that support scrapping or processing.
- Configuration & Environment: Files like .env.example and requirements.txt ensure that environment settings (like database credentials) and project dependencies are clearly defined.
- External Dependencies: The README mentions that successful usage may rely on advanced web knowledge and browser extensions for managing cookies (for example, to retrieve last_store_visited and flipp-store-code details). Additionally, a PostgreSQL database is expected as the data store, with its schema outlined in the README.

##### Establish Relationships and Interactions
a. Data Acquisition: The scrapers (main.py and run_batch.py) trigger web scraping from external grocery websites.
b. Data Processing: Once raw data is captured, it flows into processing modules (analyze_flyers.py and convert.py) where the information is analyzed and converted into a usable format.
c. Data Purging: The purge.py module is involved in cleaning the dataset – ensuring duplicates are eliminated as dictated by the SQL constraints.
d. Data Storage: The processed data is then persisted into a PostgreSQL database (using the provided "grocery" table schema).
f. Utility Interactions: The utils folder (with ref.csv) provides supporting data that may feed into the processing or analysis stages.

##### Architectural Patterns
- Modular Scripting: The separation of the main executable scripts from specific processing functions shows a clear separation of concerns.
- Configurability: The use of .env.example indicates environment-specific configuration management.

##### Specific Technologies and Design Elements
- Python is the core language, with libraries outlined in requirements.txt.
- PostgreSQL is used as the database for storing structured grocery data.
- The script interacts with external web services (grocery sites) – illustrate these as external systems.
- If applicable, mention any HTTP libraries or scraping tools being used (even though they might not be explicit in the file names, they are implied by the web-scraping purpose).

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Source
    A1["External Grocery Websites"]:::external

    %% Entry Points & Controllers
    subgraph "Entry Points & Controllers"
        B1["main.py"]:::controllers
        B2["run_batch.py"]:::controllers
        B3["legacy_code.py"]:::legacy
    end

    %% Data Processing Modules
    subgraph "Data Processing Modules"
        C1["analyze_flyers.py"]:::processing
        C2["convert.py"]:::processing
        C3["purge.py"]:::processing
    end

    %% Utility Layer
    subgraph "Utility Layer"
        D1["Utils (ref.csv)"]:::utility
    end

    %% Configuration & Environment
    subgraph "Configuration & Environment"
        E1[".env.example"]:::config
        E2["requirements.txt"]:::config
    end

    %% Data Storage
    F1["PostgreSQL Database (grocery table)"]:::database

    %% Data Flow Connections
    A1 -->|"DataFetch"| B1
    A1 -->|"DataFetch"| B2
    A1 -->|"(Legacy)DataFetch"| B3

    B1 -->|"Process"| C1
    B1 -->|"Process"| C2
    B2 -->|"Process"| C1
    B2 -->|"Process"| C2

    C1 -->|"Clean"| C3
    C2 -->|"Clean"| C3

    C3 -->|"StoreData"| F1

    D1 -->|"Support"| C1
    D1 -->|"Support"| C2
    D1 -->|"Support"| C3

    E1 ---|"Config"| B1
    E1 ---|"Config"| B2
    E2 ---|"Dependencies"| B1
    E2 ---|"Dependencies"| B2

    B3 ---|"LegacyFlow"| F1

    %% Click Events
    click B1 "https://github.com/friendlyuser/flipp_flyer_parser/blob/main/main.py"
    click B2 "https://github.com/friendlyuser/flipp_flyer_parser/blob/main/run_batch.py"
    click B3 "https://github.com/friendlyuser/flipp_flyer_parser/blob/main/legacy_code.py"
    click C1 "https://github.com/friendlyuser/flipp_flyer_parser/blob/main/analyze_flyers.py"
    click C2 "https://github.com/friendlyuser/flipp_flyer_parser/blob/main/convert.py"
    click C3 "https://github.com/friendlyuser/flipp_flyer_parser/blob/main/purge.py"
    click D1 "https://github.com/friendlyuser/flipp_flyer_parser/tree/main/utils/"
    click E1 "https://github.com/friendlyuser/flipp_flyer_parser/blob/main/.env.example"
    click E2 "https://github.com/friendlyuser/flipp_flyer_parser/blob/main/requirements.txt"

    %% Styles
    classDef external fill:#f48fb1,stroke:#333,stroke-width:2px;
    classDef controllers fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef processing fill:#ffa726,stroke:#333,stroke-width:2px;
    classDef utility fill:#ffff99,stroke:#333,stroke-width:2px;
    classDef config fill:#d1c4e9,stroke:#333,stroke-width:2px;
    classDef database fill:#90ee90,stroke:#333,stroke-width:2px;
    classDef legacy fill:#eeeeee,stroke:#333,stroke-width:2px,stroke-dasharray: 5, 5;
```
