How ssinix/crag-to-competition works
Project Type and Purpose
• This is a purely batch-oriented data pipeline and analytics project built in Python using Jupyter notebooks. Its aim is to collect climbing performance data from two external web platforms (IFSC competitions and 8a.nu profiles), merge the results into a unified dataset, and perform exploratory data analysis and hypothesis testing on that combined data.
File Structure Analysis
• Top-level directory:
• .gitignore
• README.md
• requirements.txt
• 01_data_collection/ (four notebooks handling scraping and merging)
• 02_data_analysis/ (two notebooks for EDA and hypothesis testing)
• data/ (raw and merged CSV files)
• Key patterns:
• ETL pipeline stages separated into data collection and data analysis.
• CSV file storage under data/ for both raw (ifsc_data, 8anu_data) and final_data.csv.
• A single Python environment specified by requirements.txt with libraries like Selenium, BeautifulSoup, pandas, NumPy, SciPy, matplotlib, etc.
README Insights
• Confirms two data sources: IFSC website (competition results) and 8a.nu (outdoor climbing logs).
• Describes the stepwise approach:
1. IFSC scraper
2. 8a.nu profile finder
3. 8a.nu profile scraper
4. Merger of both data streams
5. EDA and hypothesis testing
a. Main Components
1. External Data Sources:
• IFSC Competition Website
• 8a.nu Climbing Profile Site
2. Data Collection Jobs (ETL Extract & Transform):
• IfscScraper Notebook (01_data_collection/1.1_ifsc_scraper.ipynb)
• ProfileFinder Notebook (01_data_collection/1.2_8anu_profile_finder.ipynb)
• ProfileScraper Notebook (01_data_collection/1.3_8anu_profile_scraper.ipynb)
3. Data Merge Job:
• MergeData Notebook (01_data_collection/1.4_merge_data.ipynb)
4. Data Storage Layers:
• Raw CSVs: data/ifsc_data/ifsc_climbers.csv, data/8anu_data/*.csv
• Processed CSV: data/final_data.csv
5. Data Analysis Jobs:
• EDA Notebook (02_data_analysis/2.1_EDA.ipynb)
• HypothesisTesting Notebook (02_data_analysis/2.2_hypothesis_testing.ipynb)
6. Execution Environment:
• Jupyter Notebook Server + Python Runtime with dependencies from requirements.txt
7. Outputs:
• Visualizations (plots embedded in notebooks)
• Statistical test results and summary tables
b. Relationships and Data Flow
c. Architectural Patterns and Principles
• ETL pipeline: separation of extraction (scrapers), transformation (profile finder logic, grade conversions), and loading (CSV outputs).
• Layered approach: raw data layer → processed data layer → analysis layer.
• Reproducibility: each stage lives in its own notebook, enabling step-by-step reruns.
d. Key Technologies
• Python libraries: Selenium (web automation), BeautifulSoup (HTML parsing), pandas (dataframes), NumPy/SciPy (statistics), matplotlib/seaborn (visualization), fuzzywuzzy or RapidFuzz (string matching).
• Jupyter Notebooks for interactive development and reporting.
• CSV files on local filesystem as the data store.
Detail Level
• Split each notebook into its logical sub-task if desired (e.g., within ProfileFinder, show fuzzy matching block).
• Optionally, include the requirements.txt as a side node to indicate dependency management.