# ssinix/crag-to-competition architecture

> Architecture diagram of the GitHub repository ssinix/crag-to-competition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ssinix/crag-to-competition
- Repository: https://github.com/ssinix/crag-to-competition

## Overview

##### Project Type and Purpose
- This is a purely batch-oriented data pipeline and analytics project built in Python using Jupyter notebooks. Its aim is to collect climbing performance data from two external web platforms (IFSC competitions and 8a.nu profiles), merge the results into a unified dataset, and perform exploratory data analysis and hypothesis testing on that combined data.

##### File Structure Analysis
- Top-level directory:
- .gitignore
- README.md
- requirements.txt
- 01_data_collection/ (four notebooks handling scraping and merging)
- 02_data_analysis/ (two notebooks for EDA and hypothesis testing)
- data/ (raw and merged CSV files)
- Key patterns:
- ETL pipeline stages separated into data collection and data analysis.
- CSV file storage under data/ for both raw (ifsc_data, 8anu_data) and final_data.csv.
- A single Python environment specified by requirements.txt with libraries like Selenium, BeautifulSoup, pandas, NumPy, SciPy, matplotlib, etc.

##### README Insights
- Confirms two data sources: IFSC website (competition results) and 8a.nu (outdoor climbing logs).
- Describes the stepwise approach:
1. IFSC scraper
2. 8a.nu profile finder
3. 8a.nu profile scraper
4. Merger of both data streams
5. EDA and hypothesis testing

a. Main Components
1. External Data Sources:
- IFSC Competition Website
- 8a.nu Climbing Profile Site
2. Data Collection Jobs (ETL Extract & Transform):
- IfscScraper Notebook (01_data_collection/1.1_ifsc_scraper.ipynb)
- ProfileFinder Notebook (01_data_collection/1.2_8anu_profile_finder.ipynb)
- ProfileScraper Notebook (01_data_collection/1.3_8anu_profile_scraper.ipynb)
3. Data Merge Job:
- MergeData Notebook (01_data_collection/1.4_merge_data.ipynb)
4. Data Storage Layers:
- Raw CSVs: data/ifsc_data/ifsc_climbers.csv, data/8anu_data/*.csv
- Processed CSV: data/final_data.csv
5. Data Analysis Jobs:
- EDA Notebook (02_data_analysis/2.1_EDA.ipynb)
- HypothesisTesting Notebook (02_data_analysis/2.2_hypothesis_testing.ipynb)
6. Execution Environment:
- Jupyter Notebook Server + Python Runtime with dependencies from requirements.txt
7. Outputs:
- Visualizations (plots embedded in notebooks)
- Statistical test results and summary tables

b. Relationships and Data Flow

c. Architectural Patterns and Principles
- ETL pipeline: separation of extraction (scrapers), transformation (profile finder logic, grade conversions), and loading (CSV outputs).
- Layered approach: raw data layer → processed data layer → analysis layer.
- Reproducibility: each stage lives in its own notebook, enabling step-by-step reruns.

d. Key Technologies
- Python libraries: Selenium (web automation), BeautifulSoup (HTML parsing), pandas (dataframes), NumPy/SciPy (statistics), matplotlib/seaborn (visualization), fuzzywuzzy or RapidFuzz (string matching).
- Jupyter Notebooks for interactive development and reporting.
- CSV files on local filesystem as the data store.

##### Detail Level
- Split each notebook into its logical sub-task if desired (e.g., within ProfileFinder, show fuzzy matching block).
- Optionally, include the requirements.txt as a side node to indicate dependency management.

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Sources
    IFSC["IFSC Competition Website"]:::external
    A8NU["8a.nu Climbing Profile Site"]:::external

    subgraph "Jupyter Notebook Server + Python Runtime"
        subgraph "Data Collection"
            IfscScraper["IfscScraper Notebook"]:::etl
            ProfileFinder["8a.nu Profile Finder Notebook"]:::etl
            ProfileScraper["8a.nu Profile Scraper Notebook"]:::etl
            MergeData["Data Merge Notebook"]:::etl
        end
        subgraph "Data Analysis"
            EDA["Exploratory Data Analysis Notebook"]:::analysis
            Hypothesis["Hypothesis Testing Notebook"]:::analysis
        end
    end

    %% Data Stores
    RawIFSC["ifsc_climbers.csv"]:::datastore
    Raw8Profiles["8anu_profiles.csv"]:::datastore
    Raw8Climb["8anu_climbing_data.csv"]:::datastore
    FinalData["final_data.csv"]:::datastore

    %% Outputs
    Output["Visualizations & Reports"]:::output

    %% Environment and Docs
    Env["requirements.txt"]:::env
    Doc["README.md"]:::doc
    Gitignore[".gitignore"]:::doc

    %% Data Flow
    IFSC -->|"HTTP Scrape"| IfscScraper
    A8NU -->|"Profile Lookup"| ProfileFinder
    A8NU -->|"Scrape Logs"| ProfileScraper

    IfscScraper -->|writes| RawIFSC
    ProfileFinder -->|writes| Raw8Profiles
    ProfileScraper -->|writes| Raw8Climb

    RawIFSC --> MergeData
    Raw8Profiles --> MergeData
    Raw8Climb --> MergeData

    MergeData -->|writes| FinalData

    FinalData --> EDA
    FinalData --> Hypothesis

    EDA -->|generates| Output
    Hypothesis -->|generates| Output

    Env -.-> IfscScraper
    Env -.-> ProfileFinder
    Env -.-> ProfileScraper
    Env -.-> MergeData
    Env -.-> EDA
    Env -.-> Hypothesis

    Doc -.-> IfscScraper
    Gitignore -.-> IfscScraper

    %% Click Events
    click IfscScraper "https://github.com/ssinix/crag-to-competition/blob/main/01_data_collection/1.1_ifsc_scraper.ipynb"
    click ProfileFinder "https://github.com/ssinix/crag-to-competition/blob/main/01_data_collection/1.2_8anu_profile_finder.ipynb"
    click ProfileScraper "https://github.com/ssinix/crag-to-competition/blob/main/01_data_collection/1.3_8anu_profile_scraper.ipynb"
    click MergeData "https://github.com/ssinix/crag-to-competition/blob/main/01_data_collection/1.4_merge_data.ipynb"
    click RawIFSC "https://github.com/ssinix/crag-to-competition/blob/main/data/ifsc_data/ifsc_climbers.csv"
    click Raw8Profiles "https://github.com/ssinix/crag-to-competition/blob/main/data/8anu_data/8anu_profiles.csv"
    click Raw8Climb "https://github.com/ssinix/crag-to-competition/blob/main/data/8anu_data/8anu_climbing_data.csv"
    click FinalData "https://github.com/ssinix/crag-to-competition/blob/main/data/final_data.csv"
    click EDA "https://github.com/ssinix/crag-to-competition/blob/main/02_data_analysis/2.1_EDA.ipynb"
    click Hypothesis "https://github.com/ssinix/crag-to-competition/blob/main/02_data_analysis/2.2_hypothesis_testing.ipynb"
    click Env "https://github.com/ssinix/crag-to-competition/blob/main/requirements.txt"
    click Doc "https://github.com/ssinix/crag-to-competition/blob/main/README.md"
    click Gitignore "https://github.com/ssinix/crag-to-competition/blob/main/.gitignore"

    %% Styles
    classDef external fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef etl fill:#FFA500,stroke:#000,stroke-width:1px
    classDef datastore fill:#90EE90,stroke:#000,stroke-width:1px,stroke-dasharray:5 5
    classDef analysis fill:#9370DB,stroke:#000,stroke-width:1px
    classDef output fill:#FFFF00,stroke:#000,stroke-width:1px
    classDef env fill:#F5F5F5,stroke:#000,stroke-width:1px,stroke-dasharray:2 2
    classDef doc fill:#D3D33,stroke:#000,stroke-width:1px,stroke-dasharray:2 2
```
