How nikvis01/mashup_hackathon_25 works
Project Type & Purpose
• A Python‐based ETL script that scrapes bull data (with Playwright), extracts info via OpenAI Vision, transforms it, and stores it in Supabase.
Key Components
• Scraper modules (BULscraper.py, NAVscraper.py)
• GPT-4 Vision API (OpenAI)
• Transformation/DB utils (db_utils.py, data*.json, columns.json)
• Orchestrator (main.py)
• Supabase (Postgres)
• Config (.env, requirements.txt)
Data Flow & Relationships
1. main.py reads URLs & env vars
2. Scraper modules fetch pages via Playwright → full‐page screenshots
3. Screenshots → GPT-4 Vision → raw JSON
4. db_utils merges/transforms JSON (field renaming, type casting)
5. Transformed records → Supabase via REST/SDK
Architectural Patterns
• Simple pipeline (Extract → Transform → Load)
• Modular separation: scraping, transformation, persistence
• External service calls (OpenAI, Supabase)