How panagiotisgia/arbitage_betting works
Main components
• Data Collection – Represent the web scraping modules that extract sports betting data from Novibet and Stoiximan websites. This includes:
• The web_scrape_functions module (with novibet_functions.py and stoiximan_functions.py) which uses Selenium for scraping.
• The web_scrape_functions_deb module (debugging notebooks for the same functions).
• Data Storage – The raw betting data saved as CSV files in the data directory.
• Data Processing and Analysis – Main processing components including:
• The main_scripts module that contains the main.ipynb notebook which orchestrates the process.
• The queries.py script which uses SQL queries to identify arbitrage opportunities based on the collected data.
• Configuration and Dependencies – files like requirements.txt provide information on the Python libraries, and a gitignore/LICENCE file manage project integration and licensing.
Relationships and interactions
• The Web Scraping layer (both production and debugging versions) extracts betting data from external websites.
• Once scraped, the data is written to CSV files in the data folder.
• The Main_scripts initiate the workflow by invoking the scraping functions and then processing the CSV files.
• The queries.py script then performs SQL queries on the processed data to detect arbitrage opportunities.
• The system flows from data extraction → data storage → data processing/querying → arbitrage output.
Architectural patterns and design principles
• Separation of Concerns: There is a clear partition between scraping (data ingestion), data storage (CSV files), and data processing (queries and calculation of arbitrage opportunities).
• Modularization: Distinct modules and folders (web_scrape_functions vs main_scripts) indicate modular coding for each domain.
• Extensibility: Debug notebooks (web_scrape_functions_deb) allow for rapid iterations and debugging, facilitating future enhancements.
Incorporate relevant technologies and tools
• Python (with Selenium for web scraping and data processing)
• SQL (for applying queries to detect arbitrage, likely via a Python library or an embedded database if adapted)
• Jupyter Notebooks (both for main orchestration and debugging)
• CSV data file format for temporary data storage.
Critical details
• Component names are clearly labeled.
• Debugging component (web_scrape_functions_deb) separately, but indicate its close relation with the main web scraping functions.