How krishivgubba/bballrefapi works
Project Type & Purpose
• The project is a public RESTful API designed specifically to provide basketball data scraped from basketball-reference.com.
• It uses Flask to serve endpoints, integrates with a PostgreSQL database (hosted on AWS RDS) for caching scraped data, and incorporates web scraping logic alongside additional features such as rate limiting and proxy rotation.
• It is intended for deployment (using Zeet) and local development.
Analyzing the File Structure
• The top-level files and directories include:
• Dockerfile, compose.yaml, and .dockerignore for containerization and deployment.
• Python modules (Player.py, Team.py, endpoints.py) that encapsulate the core functionality:
• Player.py contains the player scraping logic.
• Team.py handles team data scraping.
• endpoints.py creates the API endpoints with Flask.
• README files outlining usage and architecture considerations.
• The presence of __pycache__ and requirements.txt confirms a Python-based project with dependency management.
Main Components of the System
• Client: External consumers (e.g., frontend applications or other services) that will eventually call the API endpoints.
• API Server:
• Built using Flask, this server in endpoints.py handles incoming GET requests for various endpoints.
• Implements endpoints such as /api/player-stats, /api/team-year-roster, and /api/player-prev-games.
• Business Logic Modules:
• Player.py: Contains classes/methods for scraping and handling player data.
• Team.py: Manages team data scraping.
• Both modules include logic to decide whether to serve data from the database or invoke the scraper.
• Database:
• A PostgreSQL instance hosted on AWS RDS that caches scraped data for throttle control and efficiency.
• Web Scraping Component:
• Invoked when the requested data is not present in the database.
• Implements rate limiting (1 request per 5 seconds) and proxy rotation to avoid getting banned by basketball-reference.com.
• Deployment & Containerization:
• Dockerfile and compose.yaml indicate containerized deployment.
Relationships and Data Flow
• Clients make GET requests to the Flask API Server.
• The API Server processes these requests in endpoints.py and, based on the request, consults the Database.
• If data exists (“cache hit”), it is returned directly; if not (“cache miss”), the API Server calls the web scraping functions from Player.py or Team.py.
• New data scraped is written back to the Database for future requests.
• The web scraping component uses proxy rotation and respects rate limits as a safeguard.
• Interactions between these components to clearly detail the decision-making process for data retrieval.
Architectural Patterns & Design Principles
• Use of the “cache aside” pattern where the database is queried first and web scraping is only triggered on a cache miss.
• Separation of concerns (API layer in endpoints.py, scraping logic in separate modules) indicating modularity.
• Use of Docker for ensuring consistency across environments and the deployment pipeline through Zeet.