# nikvis01/mashup_hackathon_25 architecture

> Architecture diagram of the GitHub repository nikvis01/mashup_hackathon_25, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nikvis01/mashup_hackathon_25
- Repository: https://github.com/nikvis01/mashup_hackathon_25

## Overview

##### Project Type & Purpose
- A Python‐based ETL script that scrapes bull data (with Playwright), extracts info via OpenAI Vision, transforms it, and stores it in Supabase.

##### Key Components
- Scraper modules (BULscraper.py, NAVscraper.py)
- GPT-4 Vision API (OpenAI)
- Transformation/DB utils (db_utils.py, data*.json, columns.json)
- Orchestrator (main.py)
- Supabase (Postgres)
- Config (.env, requirements.txt)

##### Data Flow & Relationships
1. main.py reads URLs & env vars
2. Scraper modules fetch pages via Playwright → full‐page screenshots
3. Screenshots → GPT-4 Vision → raw JSON
4. db_utils merges/transforms JSON (field renaming, type casting)
5. Transformed records → Supabase via REST/SDK

##### Architectural Patterns
- Simple pipeline (Extract → Transform → Load)
- Modular separation: scraping, transformation, persistence
- External service calls (OpenAI, Supabase)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Configuration"
        CONFIG["Config (.env, requirements.txt)"]:::config
        GITIGNORE[".gitignore"]:::config
    end

    subgraph "Orchestrator"
        MAIN["main.py"]:::orchestrator
    end

    subgraph "Extract Stage"
        BUL["BULscraper.py"]:::scraper
        NAV["NAVscraper.py"]:::scraper
        API["GPT-4 Vision API"]:::api
    end

    subgraph "Transform Stage"
        DBU["db_utils.py"]:::transform
        COL["columns.json"]:::config
        DATA1["data.json"]:::config
        DATA2["data2.json"]:::config
    end

    subgraph "Load Stage"
        SUPABASE["Supabase Postgres"]:::db
    end

    CONFIG -->|"reads config"| MAIN
    GITIGNORE ---|".env excluded"| MAIN

    MAIN -->|"invoke scrape"| BUL
    MAIN -->|"invoke scrape"| NAV

    BUL -->|"vision-OCR"| API
    NAV -->|"vision-OCR"| API

    API -->|"raw JSON output"| DBU

    COL -->|"schema definitions"| DBU
    DATA1 -->|"sample data"| DBU
    DATA2 -->|"sample data"| DBU

    DBU -->|"insert"| SUPABASE

    click CONFIG "https://github.com/nikvis01/mashup_hackathon_25/blob/main/requirements.txt"
    click GITIGNORE "https://github.com/nikvis01/mashup_hackathon_25/blob/main/.gitignore"
    click MAIN "https://github.com/nikvis01/mashup_hackathon_25/blob/main/main.py"
    click BUL "https://github.com/nikvis01/mashup_hackathon_25/blob/main/BULscraper.py"
    click NAV "https://github.com/nikvis01/mashup_hackathon_25/blob/main/NAVscraper.py"
    click DBU "https://github.com/nikvis01/mashup_hackathon_25/blob/main/db_utils.py"
    click COL "https://github.com/nikvis01/mashup_hackathon_25/blob/main/columns.json"
    click DATA1 "https://github.com/nikvis01/mashup_hackathon_25/blob/main/data.json"
    click DATA2 "https://github.com/nikvis01/mashup_hackathon_25/blob/main/data2.json"

    classDef config fill:#E0F7FA,stroke:#006064,color:#004D40;
    classDef orchestrator fill:#E8F5E9,stroke:#1B5E20,color:#2E7D32;
    classDef scraper fill:#FCE4EC,stroke:#880E4F,color:#AD1457;
    classDef api fill:#FFF3E0,stroke:#E65100,color:#EF6C00;
    classDef transform fill:#FFFDE7,stroke:#F9A825,color:#F57F17;
    classDef db fill:#ECEFF1,stroke:#37474F,color:#546E7A;
```
