How bigbrainsclub/cookie_checker works
Project Type & Purpose
• This is a backend CLI application (no GUI) whose purpose is to:
• Load a list of “services” (URLs + cookie domains + parsing rules) from config.json
• Optionally use a proxy list (proxy.txt)
• Spawn multiple threads to send HTTP requests to each service
• Validate and parse the response (regex/HTML parser)
• Optionally fetch a site logo
• Store results either in an SQLite database or in output files
Top‐Level Components
A. Configuration Loader
• Reads config.json (service definitions, proxy settings, threads, output mode)
• Reads proxy.txt when proxy_path is set
B. Main Orchestrator (main.py)
• Initializes logging, thread pool, and shared queues
• Dispatches “service” tasks
C. Thread / Task Scheduler
• Maintains a pool of worker threads (size = threads)
• Pulls tasks off the queue and invokes the service pipeline
D. Service Pipeline (in source/services/) – each pipeline stage is a separate module:
1. Request Service (request.py)
• Issues HTTP GET/POST (with or without proxy)
2. Cookie Formatter (cookie_format.py)
• Extracts / normalizes cookies if cookie_name is configured
3. Checker (checker.py)
• Performs validation of response (status codes, existence of patterns)
4. Data Parser (embedded in checker / crud)
• Applies regex or HTML parsing per parse_data rules
5. Logo Fetcher (logo.py)
• Optionally retrieves site favicon/logo via extra HTTP request
6. CRUD Service (crud.py)
• Prepares SQL statements for SQLite or structures data records
7. Writer Service (writer.py)
• Writes results:
– To SQLite via source/database.py + models.py
– To output files (filename derived from URL)
E. Persistence
• SQLite DB (via source/database.py + models.py) or
• File system (plain text or JSON files)
F. External Systems
• Target HTTP endpoints (the “services” you check)
• (Optional) Proxy servers
Relationships & Data Flow
1. Startup: main.py → Config Loader → proxy.txt & config.json
2. Task Queue: main.py → Scheduler → Worker Threads
3. For each service:
a. Request Service → HTTP Endpoint (over proxy if configured)
b. Response → Cookie Formatter → Checker → Data Parser
c. Checker/Parser output → CRUD Service → Writer Service
d. Writer Service → SQLite or Filesystem
e. Logo Fetcher may branch off after Request Service to fetch site logo
Architectural Patterns & Principles
• Modular service pipeline (single‐responsibility modules)
• Strategy pattern for parsing (regex vs. HTML parser)
• Factory pattern in Writer Service to select DB or file output
• Thread pool for concurrency
• Configuration‐driven behavior (services list in config.json)