keyfive5/cmpmapper

1. What this repo is

CMP Mapper is a Python prototype for detecting cookie-consent platforms on websites and turning those findings into Consent O Matic rules. The architecture is centered on a pipeline that goes from page collection, to banner analysis, to rule generation, to validation.

2. Main runtime surfaces

There are several entrypoints for the same core capability: command-line scripts like main.py, cmp_mapper_pro.py, web_ui.py, and demo_ui.py, plus a small API surface under api/index.py. The presence of templates/, Procfile, runtime.txt, railway.json, render.yaml, and vercel.json shows it is designed to run as a Python web service and also be deployed in serverless or PaaS environments.

3. Data collection boundary

src/collectors/ contains the ingestion layer: web_scraper.py and banner_collector.py gather HTML, JS, and banner-related page content from target sites. This is the boundary between external websites and the internal analysis pipeline.

4. Feature extraction layer

src/extractors/ turns raw page content into structured signals, such as banner structure, button labels, and CSS selectors. This layer isolates DOM parsing and selector discovery from downstream detection logic.

5. Detection and fingerprinting

src/detectors/ is the core inference subsystem. It includes banner detection, CMP fingerprinting, confidence scoring, and pattern matching, which together decide whether a page contains a consent manager and what kind it is.

6. Pattern analysis

src/analyzers/pattern_analyzer.py and top-level scripts like pharmacy_pattern_analyzer.py appear to support deeper heuristic analysis of consent UI patterns across sites. This is where reusable fingerprints and site-specific variants are identified before rule generation.

7. LLM-assisted enrichment

src/llm/ adds optional AI support for selector extraction and banner interpretation through prompt_builder.py, banner_analyzer.py, and selector_extractor.py. This is an assistive layer, not the primary pipeline, and likely helps recover selectors or interpret ambiguous banners.

8. Rule generation

src/generators/ converts detection output into Consent O Matic-compatible rules. rule_generator.py, consent_o_matic_json_generator.py, consent_o_matic_adapter.py, and template_builder.py form the transformation boundary from analysis results into deployable JSON rule templates.

9. Rule storage and artifact layout

Generated and curated outputs live in data/rules/, data/consent_o_matic_rules/, and custom-consent-o-matic-rules/rules/. The rules.schema.json file and the repo of custom rules indicate a structured rule format that is validated and reused across sites, while data/examples/ and data/consent_o_matic_research/ hold sample inputs and research artifacts.

10. Validation and test harness

src/testers/ contains the verification layer: rule testing, banner validation, and test orchestration. This is supported by many top-level test_*.py and debug_*.py scripts, which suggests the project is heavily exploratory and relies on site-specific regression testing.

11. UI and operator workflow

The web UI files (templates/index.html, web_ui.py, web_ui_fixed.py, web_ui_working.py, demo_ui.py) indicate an operator-facing workflow for single-site analysis, batch URL testing, Excel/CSV upload, and rule management. The docs in UI_README.md, QUICK_START.md, TESTING_GUIDE.md, and WORKFLOW_DIAGRAM.txt reinforce that the UI is a control plane around the same analysis engine.

12. Persistence and utilities

src/database/rule_database.py suggests a lightweight local rule store or index, while src/utils/error_handler.py provides shared error handling across the pipeline. The many one-off helper scripts (create_*, fix_*, improve_*, restore_original.py) show a repo optimized for iterative rule creation and correction rather than a single fixed production workflow.

13. External integration and deployment

The project depends on standard Python packaging (requirements.txt, requirements-vercel.txt) and is prepared for multiple deployment targets. The architecture is therefore split between a local analysis engine and a deployable web layer, with the generated Consent O Matic rules as the main output artifact consumed outside the repo.