# keyfive5/cmpmapper architecture

> Architecture diagram of the GitHub repository keyfive5/cmpmapper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/keyfive5/cmpmapper
- Repository: https://github.com/keyfive5/cmpmapper

## Overview

1. **What this repo is**
   CMP Mapper is a Python prototype for detecting cookie-consent platforms on websites and turning those findings into **Consent O Matic** rules. The architecture is centered on a pipeline that goes from page collection, to banner analysis, to rule generation, to validation.

2. **Main runtime surfaces**
   There are several entrypoints for the same core capability: command-line scripts like `main.py`, `cmp_mapper_pro.py`, `web_ui.py`, and `demo_ui.py`, plus a small API surface under `api/index.py`. The presence of `templates/`, `Procfile`, `runtime.txt`, `railway.json`, `render.yaml`, and `vercel.json` shows it is designed to run as a Python web service and also be deployed in serverless or PaaS environments.

3. **Data collection boundary**
   `src/collectors/` contains the ingestion layer: `web_scraper.py` and `banner_collector.py` gather HTML, JS, and banner-related page content from target sites. This is the boundary between external websites and the internal analysis pipeline.

4. **Feature extraction layer**
   `src/extractors/` turns raw page content into structured signals, such as banner structure, button labels, and CSS selectors. This layer isolates DOM parsing and selector discovery from downstream detection logic.

5. **Detection and fingerprinting**
   `src/detectors/` is the core inference subsystem. It includes banner detection, CMP fingerprinting, confidence scoring, and pattern matching, which together decide whether a page contains a consent manager and what kind it is.

6. **Pattern analysis**
   `src/analyzers/pattern_analyzer.py` and top-level scripts like `pharmacy_pattern_analyzer.py` appear to support deeper heuristic analysis of consent UI patterns across sites. This is where reusable fingerprints and site-specific variants are identified before rule generation.

7. **LLM-assisted enrichment**
   `src/llm/` adds optional AI support for selector extraction and banner interpretation through `prompt_builder.py`, `banner_analyzer.py`, and `selector_extractor.py`. This is an assistive layer, not the primary pipeline, and likely helps recover selectors or interpret ambiguous banners.

8. **Rule generation**
   `src/generators/` converts detection output into Consent O Matic-compatible rules. `rule_generator.py`, `consent_o_matic_json_generator.py`, `consent_o_matic_adapter.py`, and `template_builder.py` form the transformation boundary from analysis results into deployable JSON rule templates.

9. **Rule storage and artifact layout**
   Generated and curated outputs live in `data/rules/`, `data/consent_o_matic_rules/`, and `custom-consent-o-matic-rules/rules/`. The `rules.schema.json` file and the repo of custom rules indicate a structured rule format that is validated and reused across sites, while `data/examples/` and `data/consent_o_matic_research/` hold sample inputs and research artifacts.

10. **Validation and test harness**
   `src/testers/` contains the verification layer: rule testing, banner validation, and test orchestration. This is supported by many top-level `test_*.py` and `debug_*.py` scripts, which suggests the project is heavily exploratory and relies on site-specific regression testing.

11. **UI and operator workflow**
   The web UI files (`templates/index.html`, `web_ui.py`, `web_ui_fixed.py`, `web_ui_working.py`, `demo_ui.py`) indicate an operator-facing workflow for single-site analysis, batch URL testing, Excel/CSV upload, and rule management. The docs in `UI_README.md`, `QUICK_START.md`, `TESTING_GUIDE.md`, and `WORKFLOW_DIAGRAM.txt` reinforce that the UI is a control plane around the same analysis engine.

12. **Persistence and utilities**
   `src/database/rule_database.py` suggests a lightweight local rule store or index, while `src/utils/error_handler.py` provides shared error handling across the pipeline. The many one-off helper scripts (`create_*`, `fix_*`, `improve_*`, `restore_original.py`) show a repo optimized for iterative rule creation and correction rather than a single fixed production workflow.

13. **External integration and deployment**
   The project depends on standard Python packaging (`requirements.txt`, `requirements-vercel.txt`) and is prepared for multiple deployment targets. The architecture is therefore split between a local analysis engine and a deployable web layer, with the generated Consent O Matic rules as the main output artifact consumed outside the repo.

Main entry points: [`src/generators/rule_generator.py`](https://github.com/keyfive5/cmpmapper/blob/main/src/generators/rule_generator.py) (Rule generator), [`src/collectors/web_scraper.py`](https://github.com/keyfive5/cmpmapper/blob/main/src/collectors/web_scraper.py) (Web scraper), [`src/extractors/banner_extractor.py`](https://github.com/keyfive5/cmpmapper/blob/main/src/extractors/banner_extractor.py) (Banner extractor), [`src/generators/consent_o_matic_json_generator.py`](https://github.com/keyfive5/cmpmapper/blob/main/src/generators/consent_o_matic_json_generator.py) (JSON builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entry points"]
  node_main_py["Main CLI<br/>cli entrypoint<br/>[main.py]"]
  node_cmp_mapper_pro_py["Pro CLI<br/>cli entrypoint<br/>[cmp_mapper_pro.py]"]
  node_api_index_py["API<br/>http entrypoint<br/>[index.py]"]
  node_web_ui_py["Web UI<br/>ui entrypoint<br/>[web_ui.py]"]
  node_demo_ui_py["Demo UI<br/>ui entrypoint<br/>[demo_ui.py]"]
  node_templates_index_html["UI template<br/>html view<br/>[index.html]"]
end

subgraph group_pipeline["Analysis pipeline"]
  node_web_scraper_py["Web scraper<br/>collector<br/>[web_scraper.py]"]
  node_banner_collector_py["Banner collector"]
  node_banner_extractor_py["Banner extractor"]
  node_button_extractor_py["Button extractor"]
  node_selector_extractor_py["Selector extractor"]
  node_banner_detector_py["Banner detector<br/>[banner_detector.py]"]
  node_cmp_fingerprinter_py["Fingerprinting<br/>detector"]
  node_confidence_calculator_py["Confidence<br/>detector"]
  node_pattern_matcher_py["Pattern matcher<br/>detector<br/>[pattern_matcher.py]"]
  node_pattern_analyzer_py["Pattern analyzer<br/>analysis"]
  node_prompt_builder_py["Prompt builder<br/>llm helper<br/>[prompt_builder.py]"]
  node_banner_analyzer_py["LLM banner<br/>llm helper<br/>[banner_analyzer.py]"]
  node_llm_selector_extractor_py["LLM selector<br/>llm helper"]
  node_rule_generator_py["Rule generator<br/>[rule_generator.py]"]
  node_com_adapter_py["COM adapter"]
  node_com_json_generator_py["JSON builder<br/>generator"]
  node_template_builder_py["Template builder<br/>generator"]
end

subgraph group_review["Review and validation"]
  node_test_runner_py["Test runner<br/>tester<br/>[test_runner.py]"]
  node_rule_tester_py["Rule tester<br/>[rule_tester.py]"]
  node_banner_validator_py["Banner validator<br/>tester"]
end

subgraph group_artifacts["Artifacts"]
  node_generated_rules["Generated rules<br/>artifact store"]
  node_curated_rules["Curated rules<br/>artifact store"]
  node_custom_rules_repo["Custom repo<br/>external rules"]
end

subgraph group_support["Support"]
  node_rule_database_py[("Rule store<br/>database<br/>[rule_database.py]")]
  node_error_handler_py["Error handler<br/>[error_handler.py]"]
end

node_main_py -->|"collect"| node_web_scraper_py
node_main_py -->|"collect"| node_banner_collector_py
node_cmp_mapper_pro_py -->|"collect"| node_web_scraper_py
node_api_index_py -->|"run pipeline"| node_rule_generator_py
node_web_ui_py -->|"renders"| node_templates_index_html
node_web_ui_py -->|"analyze"| node_web_scraper_py
node_demo_ui_py -->|"analyze"| node_web_scraper_py
node_web_scraper_py -->|"raw html"| node_banner_extractor_py
node_banner_collector_py -->|"evidence"| node_banner_extractor_py
node_banner_extractor_py -->|"signals"| node_button_extractor_py
node_banner_extractor_py -->|"selectors"| node_selector_extractor_py
node_button_extractor_py -->|"labels"| node_pattern_matcher_py
node_selector_extractor_py -->|"selectors"| node_pattern_matcher_py
node_pattern_matcher_py -->|"match"| node_banner_detector_py
node_banner_detector_py -->|"classify"| node_cmp_fingerprinter_py
node_cmp_fingerprinter_py -->|"score"| node_confidence_calculator_py
node_pattern_analyzer_py -->|"heuristics"| node_rule_generator_py
node_prompt_builder_py -->|"prompts"| node_banner_analyzer_py
node_prompt_builder_py -->|"prompts"| node_llm_selector_extractor_py
node_banner_analyzer_py -->|"enrich"| node_rule_generator_py
node_llm_selector_extractor_py -->|"enrich"| node_rule_generator_py
node_confidence_calculator_py -->|"confidence"| node_com_adapter_py
node_rule_generator_py -->|"transform"| node_com_adapter_py
node_com_adapter_py -->|"serialize"| node_com_json_generator_py
node_com_json_generator_py -->|"template"| node_template_builder_py
node_rule_generator_py -->|"writes"| node_generated_rules
node_rule_generator_py -->|"updates"| node_curated_rules
node_com_json_generator_py -->|"exports"| node_custom_rules_repo
node_test_runner_py -->|"orchestrates"| node_rule_tester_py
node_test_runner_py -->|"orchestrates"| node_banner_validator_py
node_rule_tester_py -->|"verifies"| node_generated_rules
node_banner_validator_py -->|"checks"| node_banner_detector_py
node_rule_database_py -->|"lookup"| node_rule_generator_py
node_error_handler_py -->|"shared"| node_web_scraper_py
node_error_handler_py -->|"shared"| node_rule_generator_py

click node_main_py "https://github.com/keyfive5/cmpmapper/blob/main/main.py"
click node_cmp_mapper_pro_py "https://github.com/keyfive5/cmpmapper/blob/main/cmp_mapper_pro.py"
click node_api_index_py "https://github.com/keyfive5/cmpmapper/blob/main/api/index.py"
click node_web_ui_py "https://github.com/keyfive5/cmpmapper/blob/main/web_ui.py"
click node_demo_ui_py "https://github.com/keyfive5/cmpmapper/blob/main/demo_ui.py"
click node_templates_index_html "https://github.com/keyfive5/cmpmapper/blob/main/templates/index.html"
click node_web_scraper_py "https://github.com/keyfive5/cmpmapper/blob/main/src/collectors/web_scraper.py"
click node_banner_collector_py "https://github.com/keyfive5/cmpmapper/blob/main/src/collectors/banner_collector.py"
click node_banner_extractor_py "https://github.com/keyfive5/cmpmapper/blob/main/src/extractors/banner_extractor.py"
click node_button_extractor_py "https://github.com/keyfive5/cmpmapper/blob/main/src/extractors/button_extractor.py"
click node_selector_extractor_py "https://github.com/keyfive5/cmpmapper/blob/main/src/extractors/selector_extractor.py"
click node_banner_detector_py "https://github.com/keyfive5/cmpmapper/blob/main/src/detectors/banner_detector.py"
click node_cmp_fingerprinter_py "https://github.com/keyfive5/cmpmapper/blob/main/src/detectors/cmp_fingerprinter.py"
click node_confidence_calculator_py "https://github.com/keyfive5/cmpmapper/blob/main/src/detectors/confidence_calculator.py"
click node_pattern_matcher_py "https://github.com/keyfive5/cmpmapper/blob/main/src/detectors/pattern_matcher.py"
click node_pattern_analyzer_py "https://github.com/keyfive5/cmpmapper/blob/main/src/analyzers/pattern_analyzer.py"
click node_prompt_builder_py "https://github.com/keyfive5/cmpmapper/blob/main/src/llm/prompt_builder.py"
click node_banner_analyzer_py "https://github.com/keyfive5/cmpmapper/blob/main/src/llm/banner_analyzer.py"
click node_llm_selector_extractor_py "https://github.com/keyfive5/cmpmapper/blob/main/src/llm/selector_extractor.py"
click node_rule_generator_py "https://github.com/keyfive5/cmpmapper/blob/main/src/generators/rule_generator.py"
click node_com_adapter_py "https://github.com/keyfive5/cmpmapper/blob/main/src/generators/consent_o_matic_adapter.py"
click node_com_json_generator_py "https://github.com/keyfive5/cmpmapper/blob/main/src/generators/consent_o_matic_json_generator.py"
click node_template_builder_py "https://github.com/keyfive5/cmpmapper/blob/main/src/generators/template_builder.py"
click node_test_runner_py "https://github.com/keyfive5/cmpmapper/blob/main/src/testers/test_runner.py"
click node_rule_tester_py "https://github.com/keyfive5/cmpmapper/blob/main/src/testers/rule_tester.py"
click node_banner_validator_py "https://github.com/keyfive5/cmpmapper/blob/main/src/testers/banner_validator.py"
click node_rule_database_py "https://github.com/keyfive5/cmpmapper/blob/main/src/database/rule_database.py"
click node_error_handler_py "https://github.com/keyfive5/cmpmapper/blob/main/src/utils/error_handler.py"
click node_generated_rules "https://github.com/keyfive5/cmpmapper/tree/main/data/consent_o_matic_rules"
click node_curated_rules "https://github.com/keyfive5/cmpmapper/tree/main/data/rules"
click node_custom_rules_repo "https://github.com/keyfive5/cmpmapper/tree/main/custom-consent-o-matic-rules/rules"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_cmp_mapper_pro_py,node_api_index_py,node_web_ui_py,node_demo_ui_py,node_templates_index_html toneBlue
class node_web_scraper_py,node_banner_collector_py,node_banner_extractor_py,node_button_extractor_py,node_selector_extractor_py,node_banner_detector_py,node_cmp_fingerprinter_py,node_confidence_calculator_py,node_pattern_matcher_py,node_pattern_analyzer_py,node_prompt_builder_py,node_banner_analyzer_py,node_llm_selector_extractor_py,node_rule_generator_py,node_com_adapter_py,node_com_json_generator_py,node_template_builder_py toneAmber
class node_test_runner_py,node_rule_tester_py,node_banner_validator_py toneMint
class node_generated_rules,node_curated_rules,node_custom_rules_repo toneRose
class node_rule_database_py,node_error_handler_py toneIndigo
```

## Components

### Entry points

CLI, API, and UI surfaces that start the workflow

- [Main CLI](https://github.com/keyfive5/cmpmapper/blob/main/main.py): cli entrypoint · Primary command-line entry for running the mapper workflow · `main.py`
- [Pro CLI](https://github.com/keyfive5/cmpmapper/blob/main/cmp_mapper_pro.py): cli entrypoint · Alternative command-line runner for the core pipeline · `cmp_mapper_pro.py`
- [API](https://github.com/keyfive5/cmpmapper/blob/main/api/index.py): http entrypoint · Minimal API surface for remote triggering or integration · `api/index.py`
- [Web UI](https://github.com/keyfive5/cmpmapper/blob/main/web_ui.py): ui entrypoint · Operator-facing web interface for site analysis and rule work · `web_ui.py`
- [Demo UI](https://github.com/keyfive5/cmpmapper/blob/main/demo_ui.py): ui entrypoint · Secondary UI surface for demonstrations or simplified runs · `demo_ui.py`
- [UI template](https://github.com/keyfive5/cmpmapper/blob/main/templates/index.html): html view · Main rendered page for the web interface · `templates/index.html`

### Analysis pipeline

Core collection, extraction, detection, and generation stages

- [Web scraper](https://github.com/keyfive5/cmpmapper/blob/main/src/collectors/web_scraper.py): collector · Fetches page content from target sites · `src/collectors/web_scraper.py`
- [Banner collector](https://github.com/keyfive5/cmpmapper/blob/main/src/collectors/banner_collector.py): collector · Collects banner-related DOM and page evidence · `src/collectors/banner_collector.py`
- [Banner extractor](https://github.com/keyfive5/cmpmapper/blob/main/src/extractors/banner_extractor.py): extractor · Turns raw page content into banner structure signals · `src/extractors/banner_extractor.py`
- [Button extractor](https://github.com/keyfive5/cmpmapper/blob/main/src/extractors/button_extractor.py): extractor · Finds consent button labels and related actions · `src/extractors/button_extractor.py`
- [Selector extractor](https://github.com/keyfive5/cmpmapper/blob/main/src/extractors/selector_extractor.py): extractor · Derives CSS selectors and targeting signals · `src/extractors/selector_extractor.py`
- [Banner detector](https://github.com/keyfive5/cmpmapper/blob/main/src/detectors/banner_detector.py): detector · Identifies consent banners and their presence · `src/detectors/banner_detector.py`
- [Fingerprinting](https://github.com/keyfive5/cmpmapper/blob/main/src/detectors/cmp_fingerprinter.py): detector · Classifies CMP patterns and platform signatures · `src/detectors/cmp_fingerprinter.py`
- [Confidence](https://github.com/keyfive5/cmpmapper/blob/main/src/detectors/confidence_calculator.py): detector · Scores detection certainty and rule reliability · `src/detectors/confidence_calculator.py`
- [Pattern matcher](https://github.com/keyfive5/cmpmapper/blob/main/src/detectors/pattern_matcher.py): detector · Matches extracted signals against known consent patterns · `src/detectors/pattern_matcher.py`
- [Pattern analyzer](https://github.com/keyfive5/cmpmapper/blob/main/src/analyzers/pattern_analyzer.py): analysis · Heuristic analysis of reusable consent UI patterns · `src/analyzers/pattern_analyzer.py`
- [Prompt builder](https://github.com/keyfive5/cmpmapper/blob/main/src/llm/prompt_builder.py): llm helper · Builds prompts for AI-assisted selector and banner interpretation · `src/llm/prompt_builder.py`
- [LLM banner](https://github.com/keyfive5/cmpmapper/blob/main/src/llm/banner_analyzer.py): llm helper · Optional AI-assisted banner interpretation · `src/llm/banner_analyzer.py`
- [LLM selector](https://github.com/keyfive5/cmpmapper/blob/main/src/llm/selector_extractor.py): llm helper · Optional AI-assisted selector recovery · `src/llm/selector_extractor.py`
- [Rule generator](https://github.com/keyfive5/cmpmapper/blob/main/src/generators/rule_generator.py): generator · Converts detection output into Consent O Matic rules · `src/generators/rule_generator.py`
- [COM adapter](https://github.com/keyfive5/cmpmapper/blob/main/src/generators/consent_o_matic_adapter.py): adapter · Maps internal results into Consent O Matic structure · `src/generators/consent_o_matic_adapter.py`
- [JSON builder](https://github.com/keyfive5/cmpmapper/blob/main/src/generators/consent_o_matic_json_generator.py): generator · Builds final Consent O Matic JSON artifacts · `src/generators/consent_o_matic_json_generator.py`
- [Template builder](https://github.com/keyfive5/cmpmapper/blob/main/src/generators/template_builder.py): generator · Creates reusable rule templates from findings · `src/generators/template_builder.py`

### Review and validation

Validation, testing, and iterative correction tools

- [Test runner](https://github.com/keyfive5/cmpmapper/blob/main/src/testers/test_runner.py): tester · Orchestrates validation and regression runs · `src/testers/test_runner.py`
- [Rule tester](https://github.com/keyfive5/cmpmapper/blob/main/src/testers/rule_tester.py): tester · Evaluates generated rules against pages or fixtures · `src/testers/rule_tester.py`
- [Banner validator](https://github.com/keyfive5/cmpmapper/blob/main/src/testers/banner_validator.py): tester · Checks whether detected banners and rules behave correctly · `src/testers/banner_validator.py`

### Artifacts

Rule stores and generated Consent O Matic outputs

- [Generated rules](https://github.com/keyfive5/cmpmapper/tree/main/data/consent_o_matic_rules): artifact store · Primary output directory for generated Consent O Matic rules · `data/consent_o_matic_rules`
- [Curated rules](https://github.com/keyfive5/cmpmapper/tree/main/data/rules): artifact store · Human-edited and research-backed rule set · `data/rules`
- [Custom repo](https://github.com/keyfive5/cmpmapper/tree/main/custom-consent-o-matic-rules/rules): external rules · Custom Consent O Matic rule repository and schema · `custom-consent-o-matic-rules/rules`

### Support

Shared storage, models, and utilities

- [Rule store](https://github.com/keyfive5/cmpmapper/blob/main/src/database/rule_database.py): database · Local persistence layer for rules and lookup data · `src/database/rule_database.py`
- [Error handler](https://github.com/keyfive5/cmpmapper/blob/main/src/utils/error_handler.py): utility · Shared error handling across the pipeline · `src/utils/error_handler.py`

## Connections

- Main CLI → Web scraper: collect · CLI starts by collecting target page content
- Main CLI → Banner collector: collect · CLI can gather banner evidence directly
- Pro CLI → Web scraper: collect · Alternative runner uses the same ingestion layer
- API → Rule generator: run pipeline · API exposes the analysis-to-rule workflow
- Web UI → UI template: renders · Web app serves the operator interface template
- Web UI → Web scraper: analyze · UI triggers site collection and inspection
- Demo UI → Web scraper: analyze · Demo UI reuses the collection stage
- Web scraper → Banner extractor: raw html · Scraped pages feed structured banner extraction
- Banner collector → Banner extractor: evidence · Collected banner data is normalized for extraction
- Banner extractor → Button extractor: signals · Banner structure informs button discovery
- Banner extractor → Selector extractor: selectors · Extraction produces selector candidates
- Button extractor → Pattern matcher: labels · Button labels contribute to pattern matching
- Selector extractor → Pattern matcher: selectors · Selector signals feed matching and fingerprinting
- Pattern matcher → Banner detector: match · Matched patterns help confirm banner detection
- Banner detector → Fingerprinting: classify · Detected banners are fingerprinted into CMP types
- Fingerprinting → Confidence: score · Fingerprint results are scored for reliability
- Pattern analyzer → Rule generator: heuristics · Pattern analysis informs rule creation
- Prompt builder → LLM banner: prompts · Prompt construction supports banner interpretation
- Prompt builder → LLM selector: prompts · Prompt construction supports selector recovery
- LLM banner → Rule generator: enrich · Optional AI findings refine generation inputs
- LLM selector → Rule generator: enrich · Recovered selectors improve generated rules
- Confidence → COM adapter: confidence · Confidence values accompany generation output
- Rule generator → COM adapter: transform · Core generation hands off to Consent O Matic adaptation
- COM adapter → JSON builder: serialize · Adapted structure becomes final JSON
- JSON builder → Template builder: template · JSON output can be wrapped into reusable templates
- Rule generator → Generated rules: writes · Primary output is stored as generated rule artifacts
- Rule generator → Curated rules: updates · Curated rule files capture refined variants
- JSON builder → Custom repo: exports · Generated JSON can be exported to the custom rule repository
- Test runner → Rule tester: orchestrates · Test runner coordinates rule validation
- Test runner → Banner validator: orchestrates · Test runner coordinates banner checks
- Rule tester → Generated rules: verifies · Tester evaluates stored generated rules
- Banner validator → Banner detector: checks · Validation compares detector output with observed banners
- Rule store → Rule generator: lookup · Local store supplies existing rules and prior findings
- Error handler → Web scraper: shared · Shared error handling supports the ingestion boundary
- Error handler → Rule generator: shared · Shared error handling supports generation and export