# bigbrainsclub/cookie_checker architecture

> Architecture diagram of the GitHub repository bigbrainsclub/cookie_checker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bigbrainsclub/cookie_checker
- Repository: https://github.com/bigbrainsclub/cookie_checker

## Overview

##### Project Type & Purpose
- This is a backend CLI application (no GUI) whose purpose is to:
- Load a list of “services” (URLs + cookie domains + parsing rules) from config.json
- Optionally use a proxy list (proxy.txt)
- Spawn multiple threads to send HTTP requests to each service
- Validate and parse the response (regex/HTML parser)
- Optionally fetch a site logo
- Store results either in an SQLite database or in output files

##### Top‐Level Components
A. Configuration Loader
- Reads config.json (service definitions, proxy settings, threads, output mode)
- Reads proxy.txt when proxy_path is set
B. Main Orchestrator (main.py)
- Initializes logging, thread pool, and shared queues
- Dispatches “service” tasks
C. Thread / Task Scheduler
- Maintains a pool of worker threads (size = threads)
- Pulls tasks off the queue and invokes the service pipeline
D. Service Pipeline (in source/services/) – each pipeline stage is a separate module:
1. Request Service (request.py)
- Issues HTTP GET/POST (with or without proxy)
2. Cookie Formatter (cookie_format.py)
- Extracts / normalizes cookies if cookie_name is configured
3. Checker (checker.py)
- Performs validation of response (status codes, existence of patterns)
4. Data Parser (embedded in checker / crud)
- Applies regex or HTML parsing per parse_data rules
5. Logo Fetcher (logo.py)
- Optionally retrieves site favicon/logo via extra HTTP request
6. CRUD Service (crud.py)
- Prepares SQL statements for SQLite or structures data records
7. Writer Service (writer.py)
- Writes results:
– To SQLite via source/database.py + models.py
– To output files (filename derived from URL)
E. Persistence
- SQLite DB (via source/database.py + models.py) or
- File system (plain text or JSON files)
F. External Systems
- Target HTTP endpoints (the “services” you check)
- (Optional) Proxy servers

##### Relationships & Data Flow
1. Startup: main.py → Config Loader → proxy.txt & config.json
2. Task Queue: main.py → Scheduler → Worker Threads
3. For each service:
a. Request Service → HTTP Endpoint (over proxy if configured)
b. Response → Cookie Formatter → Checker → Data Parser
c. Checker/Parser output → CRUD Service → Writer Service
d. Writer Service → SQLite or Filesystem
e. Logo Fetcher may branch off after Request Service to fetch site logo

##### Architectural Patterns & Principles
- Modular service pipeline (single‐responsibility modules)
- Strategy pattern for parsing (regex vs. HTML parser)
- Factory pattern in Writer Service to select DB or file output
- Thread pool for concurrency
- Configuration‐driven behavior (services list in config.json)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        ConfigLoader["Configuration Loader"]:::orchestration
        Scheduler["Thread / Task Scheduler"]:::orchestration
        WorkerThreads["Worker Threads Pool"]:::orchestration
    end

    %% Service Pipeline
    subgraph "Service Pipeline"
        direction TB
        RequestService["Request Service"]:::services
        CookieFormatter["Cookie Formatter"]:::services
        CheckerService["Checker / Data Parser"]:::services
        CRUDService["CRUD Service"]:::services
        WriterService["Writer Service"]:::services
        LogoFetcher["Logo Fetcher"]:::services
    end

    %% Persistence Layer
    subgraph "Persistence"
        direction TB
        SQLiteDB["SQLite DB Handler"]:::storage
        Models["ORM / Data Models"]:::storage
        Filesystem["Filesystem Output"]:::storage
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        HTTPEndpoints["HTTP Endpoints"]:::external
        ProxyServers["Proxy Servers"]:::external
    end

    %% Project Metadata
    ProjectMeta["Project Config (pyproject.toml, requirements.txt)"]:::orchestration

    %% Data Flows
    ProjectMeta -->|"load config & metadata"| ConfigLoader
    ConfigLoader -->|"ServiceDefinitions"| Scheduler
    ConfigLoader -->|"ProxyList"| ProxyServers
    Scheduler -->|"enqueue tasks"| WorkerThreads
    WorkerThreads -->|"ServiceTask"| RequestService

    RequestService -->|"HTTP Request"| HTTPEndpoints
    ProxyServers -.->|"via proxy"| RequestService
    RequestService -->|"HTTP Response"| CookieFormatter
    CookieFormatter -->|"WithCookies"| CheckerService
    CheckerService -->|"ParsedData"| CRUDService
    CheckerService -.->|"Logo URL"| LogoFetcher
    LogoFetcher -.->|"Fetch Logo"| HTTPEndpoints
    CRUDService -->|"DataRecord"| WriterService
    WriterService -->|"SQL Statements"| SQLiteDB
    WriterService -->|"Data Models"| Models
    WriterService -->|"Output Files"| Filesystem

    %% Click Events
    click ConfigLoader "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/config.py"
    click Scheduler "https://github.com/bigbrainsclub/cookie_checker/blob/main/main.py"
    click WorkerThreads "https://github.com/bigbrainsclub/cookie_checker/blob/main/main.py"
    click RequestService "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/services/request.py"
    click CookieFormatter "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/services/cookie_format.py"
    click CheckerService "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/services/checker.py"
    click CRUDService "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/services/crud.py"
    click WriterService "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/services/writer.py"
    click LogoFetcher "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/services/logo.py"
    click SQLiteDB "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/database.py"
    click Models "https://github.com/bigbrainsclub/cookie_checker/blob/main/source/models.py"
    click ProjectMeta "https://github.com/bigbrainsclub/cookie_checker/blob/main/pyproject.toml"

    %% Styles
    classDef orchestration fill:#add8e6,stroke:#333,stroke-width:1px
    classDef services fill:#90ee90,stroke:#333,stroke-width:1px
    classDef external fill:#ffff99,stroke:#333,stroke-width:1px
    classDef storage fill:#ffa500,stroke:#333,stroke-width:1px
```
