How masnat/pokedex-api works
Project Type and Purpose
• This project is a Python-based web application that exposes a REST API to serve Pokémon data.
• It is composed of two main functionalities: crawling data from an external Pokémon site and then serving that data via a RESTful API.
• The project interacts with a PostgreSQL database which stores Pokémon information.
Main Components of the System
• Data Crawler & ETL Scripts:
• Files: crawl_pokedex.py, generate_table.py, and truncate_table.py.
• Purpose: Creating the required database schema, populating it with data crawled from an external source (the Pokémon Dex website), and the ability to truncate/refresh the data.
• Database Connector:
• File: conn.py.
• Purpose: Contains the database configuration and connection logic for PostgreSQL.
• REST API Server:
• File: server.py.
• Purpose: Runs the web server that exposes the REST API endpoints for listing all Pokémon, querying specific details, or retrieving a random Pokémon.
• Database (External Component):
• PostgreSQL database named “pokedex” storing the Pokémon data.
Relationships and Interactions
• The "generate_table.py" script is executed to create the necessary tables in the PostgreSQL database.
• The "crawl_pokedex.py" script uses the database connector (conn.py) to populate the tables by scraping the external Pokémon Dex website.
• "truncate_table.py" provides maintenance capability if data reset/cleanup is needed.
• The "server.py" script starts the REST API, which connects (using conn.py’s configuration) to the PostgreSQL database. The API handles various query parameters and routes to deliver Pokémon data.
• External Source: The crawler fetches data from the public "Pokemon Dex" website.
Architectural Patterns and Design Principles
• Layered Architecture:
• Data Access Layer: Database connector (conn.py) and the underlying PostgreSQL.
• Business/Data Processing Layer: Data crawling and table management scripts.
• Presentation/API Layer: REST API server (server.py).
• Clear separation of concerns is maintained between data ingestion (crawling and table generation) and data serving (API).
• Use of external libraries and tools such as Requests for HTTP queries, BeautifulSoup for parsing HTML, SQLAlchemy for database interactions, and psycopg2 as the PostgreSQL driver.