# greco1899/scrape_ufc_stats architecture

> Architecture diagram of the GitHub repository greco1899/scrape_ufc_stats, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/greco1899/scrape_ufc_stats
- Repository: https://github.com/greco1899/scrape_ufc_stats

## Overview

##### Project type and purpose
- This is a data‑pipeline / ETL‑style Python scraper project whose sole purpose is to fetch historical and incremental UFC events, fights, stats, and fighter “tale of the tape” data from ufcstats.com and store it in CSV files for downstream analysis or ML.
- There is no traditional frontend; it is purely a backend/scraper service, orchestrated via notebooks and scripts, with scheduled runs on PythonAnywhere.

##### File structure
- Top‑level files:
- scrape_ufc_stats_library.py: core scraping and parsing logic encapsulated as reusable functions
- multiple Jupyter notebooks (*.ipynb) illustrating full‑historical and single‑run workflows
- scrape_ufc_stats_unparsed_data.py: incremental refresh script
- scrape_ufc_stats_config.yaml: centralized configuration for URLs, output directories, etc.
- requirements.txt: Python dependencies
- Data output CSVs live at the repo root after each run.
- No database—flat files only.
- Scheduler: described in the README as running daily on PythonAnywhere.

##### README for additional insights
- Describes two modes:
a) Full‑historical scrape (hours‑long notebooks)
b) Incremental daily refresh (script runs in minutes)
- Notes use of PythonAnywhere’s scheduler to automatically execute scrape_ufc_stats_unparsed_data.py and push updated CSVs back to GitHub.
- Future roadmap: cleaning, exploration, ML prediction—but out of scope for now.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Source Layer"
        UFS["ufcstats.com"]:::external
    end

    subgraph "Processing Layer"
        SL["Scraper Library"]:::library
        CFG["Configuration (YAML)"]:::config
        subgraph "Notebooks (Full Historical)"
            NB1["All Historical Data Notebook"]:::notebook
            NB2["Fighter Tale of the Tape Notebook"]:::notebook
            NB3["Working Example Notebook"]:::notebook
        end
        NB4["Incremental Refresh Notebook"]:::notebook
        SCRIP["Incremental Refresh Script"]:::script
    end

    subgraph "Storage & Orchestration Layer"
        subgraph "CSV Data Files"
            EVT["ufc_event_details.csv"]:::datastore
            FDT["ufc_fight_details.csv"]:::datastore
            FRS["ufc_fight_results.csv"]:::datastore
            FST["ufc_fight_stats.csv"]:::datastore
            FDTL["ufc_fighter_details.csv"]:::datastore
            TOT["ufc_fighter_tott.csv"]:::datastore
        end
        GH["GitHub Repo"]:::orchestration
        SCH["PythonAnywhere Scheduler"]:::orchestration
        DEV["Developer Machine"]:::orchestration
        subgraph "Repository Files"
            REQ["requirements.txt"]:::doc
            RMD["README.md"]:::doc
            GITI[".gitignore"]:::doc
            LIC["LICENSE"]:::doc
        end
    end

    UFS -->|HTTP GET| SL
    SL -->|parse & normalize| EVT
    SL --> FDT
    SL --> FRS
    SL --> FST
    SL --> FDTL
    SL --> TOT

    SCH -->|triggers| SCRIP
    SCRIP -->|calls| SL
    SCRIP -->|uses| CFG
    SCRIP -->|writes| EVT
    SCRIP --> FDT
    SCRIP --> FRS
    SCRIP --> FST
    SCRIP --> FDTL
    SCRIP --> TOT
    SCRIP -->|push updates| GH

    DEV -->|runs| NB1
    DEV --> NB2
    DEV --> NB3
    DEV --> NB4
    DEV -->|edits| SL
    DEV --> CFG

    NB1 -->|calls| SL
    NB1 --> CFG
    NB2 --> SL
    NB2 --> CFG
    NB3 --> SL
    NB3 --> CFG
    NB4 --> SL
    NB4 --> CFG

    GH -->|hosts code & CSVs| DEV
    GH -->|source for| SCH

    click SL "https://github.com/greco1899/scrape_ufc_stats/blob/main/scrape_ufc_stats_library.py"
    click CFG "https://github.com/greco1899/scrape_ufc_stats/blob/main/scrape_ufc_stats_config.yaml"
    click SCRIP "https://github.com/greco1899/scrape_ufc_stats/blob/main/scrape_ufc_stats_unparsed_data.py"
    click NB4 "https://github.com/greco1899/scrape_ufc_stats/blob/main/scrape_ufc_stats_unparsed_data.ipynb"
    click NB1 "https://github.com/greco1899/scrape_ufc_stats/blob/main/scrape_ufc_stats_all_historical_data.ipynb"
    click NB2 "https://github.com/greco1899/scrape_ufc_stats/blob/main/scrape_ufc_stats_fighter_tott.ipynb"
    click NB3 "https://github.com/greco1899/scrape_ufc_stats/blob/main/scrape_ufc_stats_working_example.ipynb"
    click REQ "https://github.com/greco1899/scrape_ufc_stats/blob/main/requirements.txt"
    click RMD "https://github.com/greco1899/scrape_ufc_stats/blob/main/README.md"
    click GITI "https://github.com/greco1899/scrape_ufc_stats/blob/main/.gitignore"
    click LIC "https://github.com/greco1899/scrape_ufc_stats/tree/main/LICENSE"
    click EVT "https://github.com/greco1899/scrape_ufc_stats/blob/main/ufc_event_details.csv"
    click FDT "https://github.com/greco1899/scrape_ufc_stats/blob/main/ufc_fight_details.csv"
    click FRS "https://github.com/greco1899/scrape_ufc_stats/blob/main/ufc_fight_results.csv"
    click FST "https://github.com/greco1899/scrape_ufc_stats/blob/main/ufc_fight_stats.csv"
    click FDTL "https://github.com/greco1899/scrape_ufc_stats/blob/main/ufc_fighter_details.csv"
    click TOT "https://github.com/greco1899/scrape_ufc_stats/blob/main/ufc_fighter_tott.csv"

    classDef external fill:#f9f,stroke:#333,stroke-width:1px
    classDef library fill:#bbf,stroke:#333,stroke-width:1px
    classDef config fill:#bfb,stroke:#333,stroke-width:1px
    classDef script fill:#ffb,stroke:#333,stroke-width:1px
    classDef notebook fill:#ffd,stroke:#333,stroke-width:1px
    classDef datastore fill:#fee,stroke:#333,stroke-width:1px
    classDef orchestration fill:#dfd,stroke:#333,stroke-width:1px
    classDef doc fill:#eee,stroke:#333,stroke-width:1px
```
