# kardeepakkumar/goodreads-advanced-search architecture

> Architecture diagram of the GitHub repository kardeepakkumar/goodreads-advanced-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kardeepakkumar/goodreads-advanced-search
- Repository: https://github.com/kardeepakkumar/goodreads-advanced-search

## Overview

##### Project type and purpose
- This is a full‑stack, monolithic web application whose primary role is to **scrape** book data from Goodreads and **filter/display** it via a browser interface.
- The README calls it “Goodreads Advanced Search” – a Flask‑based scraper + filter tool that persists data in a JSON Lines file and offers a simple frontend for queries.

##### File structure
Top‑level files and folders:
- app.py – the Flask entrypoint handling HTTP routes
- scraper.py – encapsulates the logic to crawl Goodreads and write to `books_raw.jl`
- filter_books.py – applies genre/min‑rating filters to the JSONL store
- convertToJson.py – utility to convert JSONL → JSON (for docs or other consumers)
- static/ + templates/ – frontend assets (HTML, CSS, JS)
- docs/ – standalone demo site version
- Dockerfile, requirements.txt, tests/ – containerization, dependencies, and test suite

Patterns:
- MVC‑ish: Flask controllers in app.py, Jinja templates for views, static scripts for client logic
- Single datastore: JSON Lines file used as a lightweight database
- Docker-based deployment

##### Supplementary insights from README
- Uses Flask to serve both API endpoints (`/scrape`, `/filter`) and static UI.
- Relies on a browser cookie (cookie.txt) for authenticated scraping.
- No external database—everything lives on disk in `books_raw.jl`.

a. Main components
1. User Browser (UI)
2. Frontend Assets
- HTML templates (index.html, filter.html, scrape.html)
- JS modules (app.js, tabs.js, pagination.js, scrape.js)
- CSS modules
3. Flask Application
- HTTP routes: GET `/` (UI), POST `/scrape`, POST `/filter`
- Controllers invoke scraper.py or filter_books.py
4. Scraper Module
- Requests to Goodreads (external service) using cookie.txt
- Parses pages, emits JSON Lines
5. Filter Module
- Reads `books_raw.jl`, applies filter criteria, returns JSON subset
6. Data Store
- `books_raw.jl` (JSON Lines file on disk)
7. External Service
- Goodreads website

b. Relationships & interactions
- User → Browser UI → invokes Flask UI route
- User clicks “Scrape” → JS calls Flask `/scrape` → Flask invokes scraper.py → HTTP requests to Goodreads → writes to JSONL → progress streamed back to JS → UI progress bar
- User selects filters → JS calls Flask `/filter` (or submits form) → Flask invokes filter_books.py → reads JSONL → returns filtered JSON → JS renders results

c. Architectural patterns & principles
- MVC separation (controllers, templates, static)
- Single‑process, file‑based datastore (no RDBMS)
- Containerized deployment via Docker

d. Key technologies
- Python / Flask
- Jinja2 templating
- JSON Lines file for storage
- Vanilla JavaScript for client–server interactions
- Docker for packaging

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Docker Container Boundary
    subgraph "Docker Container" 
        direction TB

        %% External Browser & UI
        subgraph "User & Browser" 
            direction TB
            Browser["User Browser"]:::external
        end

        %% Frontend Layer
        subgraph "Frontend Layer" 
            direction TB
            subgraph Templates 
                direction TB
                IndexHTML["index.html"]:::frontend
                FilterHTML["filter.html"]:::frontend
                ScrapeHTML["scrape.html"]:::frontend
            end
            subgraph StaticAssets
                direction TB
                subgraph JSModules
                    direction TB
                    AppJS["app.js"]:::frontend
                    TabsJS["tabs.js"]:::frontend
                    PaginationJS["pagination.js"]:::frontend
                    ScrapeJS["scrape.js"]:::frontend
                end
                subgraph CSSStyles
                    direction TB
                    MainCSS["main.css"]:::frontend
                    FilterCSS["filter.css"]:::frontend
                    ScrapeCSS["scrape.css"]:::frontend
                end
            end
        end

        %% Backend Layer
        subgraph "Backend Layer" 
            direction TB
            FlaskApp["Flask Application"]:::backend
            Scraper["Scraper Module"]:::backend
            Filter["Filter Module"]:::backend
            Converter["JSONL→JSON Utility"]:::backend
            Cookie["cookie.txt"]:::backend
        end

        %% Data Store
        DataStore["books_raw.jl"]:::datastore

        %% External Service
        Goodreads["Goodreads Website"]:::external

    end

    %% Connections
    Browser -->|"GET /"| FlaskApp
    FlaskApp -->|"renders Templates"| IndexHTML
    FlaskApp -->|"renders Templates"| FilterHTML
    FlaskApp -->|"renders Templates"| ScrapeHTML

    Browser -->|"loads JS/CSS"| AppJS
    Browser --> AppJS
    Browser --> TabsJS
    Browser --> PaginationJS
    Browser --> ScrapeJS
    Browser --> MainCSS
    Browser --> FilterCSS
    Browser --> ScrapeCSS

    AppJS -->|"POST /scrape"| FlaskApp
    AppJS -->|"POST /filter"| FlaskApp

    FlaskApp -->|"invokes"| Scraper
    FlaskApp -->|"invokes"| Filter

    Scraper -->|"reads cookie"| Cookie
    Scraper -->|"HTTP requests"| Goodreads
    Scraper -->|"writes JSONL"| DataStore

    Filter -->|"reads JSONL"| DataStore
    Filter -->|"returns JSON"| FlaskApp

    Converter -->|"reads JSONL"| DataStore
    Converter -->|"writes JSON"| Converter

    %% Click Events
    click FlaskApp "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/app.py"
    click Scraper "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/scraper.py"
    click Filter "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/filter_books.py"
    click Converter "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/convertToJson.py"
    click IndexHTML "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/templates/index.html"
    click FilterHTML "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/templates/filter.html"
    click ScrapeHTML "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/templates/scrape.html"
    click AppJS "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/static/scripts/app.js"
    click TabsJS "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/static/scripts/tabs.js"
    click PaginationJS "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/static/scripts/pagination.js"
    click ScrapeJS "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/static/scripts/scrape.js"
    click MainCSS "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/static/styles/main.css"
    click FilterCSS "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/static/styles/filter.css"
    click ScrapeCSS "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/static/styles/scrape.css"
    click Dockerfile "https://github.com/kardeepakkumar/goodreads-advanced-search/tree/main/Dockerfile"
    click Requirements "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/requirements.txt"
    click ReqTest "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/requirements-testing.txt"
    click TestConftest "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/tests/conftest.py"
    click TestScraper "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/tests/test_scraper.py"
    click TestFilter "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/tests/test_filter.py"
    click TestUI "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/tests/test_ui.py"
    click DocsIndex "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/docs/index.html"
    click DocsJSON "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/docs/books.json"
    click DocsScript "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/docs/script.js"
    click DocsStyles "https://github.com/kardeepakkumar/goodreads-advanced-search/blob/main/docs/styles.css"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef backend fill:#d4edda,stroke:#333,stroke-width:1px
    classDef external fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef datastore fill:#e2e3e5,stroke:#333,stroke-width:1px
```
