# city-of-helsinki/gdpr-compliance-scanner architecture

> Architecture diagram of the GitHub repository city-of-helsinki/gdpr-compliance-scanner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/city-of-helsinki/gdpr-compliance-scanner
- Repository: https://github.com/city-of-helsinki/gdpr-compliance-scanner

## Overview

Key Components to Include:

##### Scanner Core Components
- It to multiple collector modules (represented as a collector subsystem)
- Compliance checker as a separate component

##### Data Collection Layer
- Storage collectors (cookies, localStorage, sessionStorage, indexedDB, cacheStorage)
- Page data collectors
- Settings collectors
Each should be represented as individual modules within the collectors group

##### Processing & Analysis
- Compliance checking module
- Rules engine connecting to the HDS Cookie Banner integration
- Error logging system

##### Output & Reporting
- Report generator component
- Static file server for report hosting
- Report assets (HTML, CSS, JS)

**[External Systems]**

- ▼
**[Scanner Engine] ─────────────┐**

- ▼                        ▼
**[Collectors Layer]    [Configuration]**

- ▼                        │
**[Compliance Checker]◄────────┘**

- ▼
**[Report Generator]**

- ▼
**[Web Server]**

- Blue: Core scanner components
- Green: Data collection modules
- Yellow: Processing & analysis
- Orange: Output & reporting
- Gray: External systems/dependencies

Specific Implementation Details:

- How data flows from the scanner engine to collectors
- How collected data passes through compliance checking
- Demonstrate report generation flow

##### External Integrations
- Connection to HDS Cookie Banner system
- Website under test
- API endpoints for settings retrieval

##### Storage Components
- cacheStorage
- cookies
- indexedDB
- localStorage
- sessionStorage

##### Report Generation Pipeline
Flow from:
- Raw data collection
- Compliance analysis
- Report generation
- Static file serving

Important Considerations:

##### Asynchronous Operations
- Parallel scanning capabilities

##### Configuration Flow
- How config/config.js influences different components
- Where URL configurations are processed

##### Error Handling
- Error logger utility
- Error flow paths

##### Monitoring & Utilities
- Timer utility
- Domain tracking functionality

- Double lines: Data flow
- Diamonds: Decision points
- Rectangles: Processing modules
- Rounded rectangles: External systems

Remember to include notes about scalability and potential bottlenecks, as this system needs to handle multiple URLs and storage types efficiently.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef collector fill:#4CAF50,stroke:#2E7D32,color:white
    classDef storage fill:#81C784,stroke:#388E3C,color:white
    classDef analysis fill:#FFC107,stroke:#FFA000,color:black
    classDef output fill:#FF9800,stroke:#F57C00,color:white
    classDef external fill:#9E9E9E,stroke:#616161,color:white
    classDef utility fill:#673AB7,stroke:#512DA8,color:white

    %% External Systems
    ExtSys["External Systems"]:::external
    HDS["HDS Cookie Banner"]:::external

    %% Core Components
    ScanEngine["Scanner Engine"]:::core
    click ScanEngine "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/index.js"

    %% Collectors Subgraph
    subgraph Collectors
        CollectorsMain["Collectors Layer"]:::collector
        click CollectorsMain "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/tree/main/src/collectors/"
        
        subgraph StorageCollectors ["Storage Collectors"]
            Cache["Cache Storage"]:::storage
            Cookies["Cookies"]:::storage
            IndexedDB["IndexedDB"]:::storage
            LocalStorage["LocalStorage"]:::storage
            SessionStorage["SessionStorage"]:::storage
            
            click Cache "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/collectors/storages/cacheStorage.js"
            click Cookies "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/collectors/storages/cookies.js"
            click IndexedDB "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/collectors/storages/indexedDB.js"
            click LocalStorage "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/collectors/storages/localStorage.js"
            click SessionStorage "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/collectors/storages/sessionStorage.js"
        end

        PageData["Page Data Collector"]:::collector
        click PageData "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/collectors/collectDataFromPages.js"

        GroupSettings["Group Settings Collector"]:::collector
        GroupHashes["Group Hashes Collector"]:::collector
        click GroupSettings "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/collectors/collectGroupSettings.js"
        click GroupHashes "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/collectors/collectGroupHashes.js"
    end

    %% Utilities
    subgraph Utilities
        ErrorLogger["Error Logger"]:::utility
        Timer["Timer"]:::utility
        DomainTracker["Domain Tracker"]:::utility
        UrlProcessor["URL Processor"]:::utility

        click ErrorLogger "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/utils/errorLogger.js"
        click Timer "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/utils/timer.js"
        click DomainTracker "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/utils/fetchTrackingDomains.js"
        click UrlProcessor "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/utils/getUrlsToScan.js"
    end

    %% Configuration
    Config["Configuration"]:::core
    click Config "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/config/config.js"

    %% Analysis Components
    ComplianceChecker["Compliance Checker"]:::analysis
    click ComplianceChecker "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/compliance/checkCompliance.js"

    %% Output Components
    ReportGen["Report Generator"]:::output
    click ReportGen "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/reporter/generateReport.js"

    WebServer["Web Server"]:::output
    click WebServer "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/src/server/server.js"

    subgraph ReportAssets ["Report Assets"]
        ReportHTML["HTML"]:::output
        ReportCSS["CSS"]:::output
        ReportJS["JavaScript"]:::output

        click ReportHTML "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/reports/report.html"
        click ReportCSS "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/reports/css/styles.css"
        click ReportJS "https://github.com/City-of-Helsinki/GDPR-compliance-scanner/blob/main/reports/js/reportFunctions.js"
    end

    %% Relationships
    ExtSys --> ScanEngine
    ScanEngine --> CollectorsMain
    ScanEngine --> Config
    
    CollectorsMain --> StorageCollectors
    CollectorsMain --> PageData
    CollectorsMain --> GroupSettings
    CollectorsMain --> GroupHashes
    
    Config -.-> ComplianceChecker
    CollectorsMain --> ComplianceChecker
    HDS -.-> ComplianceChecker
    
    ComplianceChecker --> ReportGen
    ReportGen --> WebServer
    ReportGen --> ReportAssets
    
    %% Utility Connections
    ErrorLogger -.-> ScanEngine
    Timer -.-> ScanEngine
    DomainTracker -.-> CollectorsMain
    UrlProcessor -.-> ScanEngine

    %% Legend
    subgraph Legend
        L1["Core Components"]:::core
        L2["Collectors"]:::collector
        L3["Storage"]:::storage
        L4["Analysis"]:::analysis
        L5["Output"]:::output
        L6["External Systems"]:::external
        L7["Utilities"]:::utility
    end
```
