How kardeepakkumar/goodreads-advanced-search works
Project type and purpose
• This is a full‑stack, monolithic web application whose primary role is to scrape book data from Goodreads and filter/display it via a browser interface.
• The README calls it “Goodreads Advanced Search” – a Flask‑based scraper + filter tool that persists data in a JSON Lines file and offers a simple frontend for queries.
File structure
Top‑level files and folders:
• app.py – the Flask entrypoint handling HTTP routes
• scraper.py – encapsulates the logic to crawl Goodreads and write to books_raw.jl
• filter_books.py – applies genre/min‑rating filters to the JSONL store
• convertToJson.py – utility to convert JSONL → JSON (for docs or other consumers)
• static/ + templates/ – frontend assets (HTML, CSS, JS)
• docs/ – standalone demo site version
• Dockerfile, requirements.txt, tests/ – containerization, dependencies, and test suite
Patterns:
• MVC‑ish: Flask controllers in app.py, Jinja templates for views, static scripts for client logic
• Single datastore: JSON Lines file used as a lightweight database
• Docker-based deployment
Supplementary insights from README
• Uses Flask to serve both API endpoints (/scrape, /filter) and static UI.
• Relies on a browser cookie (cookie.txt) for authenticated scraping.
• No external database—everything lives on disk in books_raw.jl.
a. Main components
1. User Browser (UI)
2. Frontend Assets
• HTML templates (index.html, filter.html, scrape.html)
• JS modules (app.js, tabs.js, pagination.js, scrape.js)
• CSS modules
3. Flask Application
• HTTP routes: GET / (UI), POST /scrape, POST /filter
• Controllers invoke scraper.py or filter_books.py
4. Scraper Module
• Requests to Goodreads (external service) using cookie.txt
• Parses pages, emits JSON Lines
5. Filter Module
• Reads books_raw.jl, applies filter criteria, returns JSON subset
6. Data Store
• books_raw.jl (JSON Lines file on disk)
7. External Service
• Goodreads website
b. Relationships & interactions
• User → Browser UI → invokes Flask UI route
• User clicks “Scrape” → JS calls Flask /scrape → Flask invokes scraper.py → HTTP requests to Goodreads → writes to JSONL → progress streamed back to JS → UI progress bar
• User selects filters → JS calls Flask /filter (or submits form) → Flask invokes filter_books.py → reads JSONL → returns filtered JSON → JS renders results
c. Architectural patterns & principles
• MVC separation (controllers, templates, static)
• Single‑process, file‑based datastore (no RDBMS)
• Containerized deployment via Docker
d. Key technologies
• Python / Flask
• Jinja2 templating
• JSON Lines file for storage
• Vanilla JavaScript for client–server interactions
• Docker for packaging