# dnswarden/blocklist-staging architecture

> Architecture diagram of the GitHub repository dnswarden/blocklist-staging, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dnswarden/blocklist-staging
- Repository: https://github.com/dnswarden/blocklist-staging

## Overview

##### Project type and purpose
- This is an open-source “staging blacklist/whitelist/customlist” repository for the dnswarden service.
- It contains raw data (blacklists, whitelists, custom filters), configuration files, a Node.js script (download.js), and GitHub Actions workflows to validate, generate, and update blocklists for dnswarden.

##### File structure
- Top-level artifacts:
- .github/workflows – CI workflows: checks, generateRegex, update
- config – JSON files (conn.json, customfilter.json, hashes.json)
- blacklist, whitelist, custom_filters – raw domain lists and custom filter blobs
- sources – metadata files (.md) describing each source list
- download.js – main Node.js script that pulls config, fetches or reads lists and produces outputs
- edit_here_to_add_blocklists.json – config for default lists used by dnswarden UI
- package.json – Node.js dependencies and scripts
- test_data – example inputs/outputs for local testing
- GitHub Actions orchestrate: linting/validation (.github/workflows/checks.yml), building regex files (generateRegex.yml), updating histories (update.yml).

##### Insights from README
- Describes how to add or remove blocklists via edit_here_to_add_blocklists.json
- Explains the schema of list entries (name, category, url[], filterType[], value, enabled)
- Points to dnswarden’s UI endpoints (customfilter.html, search.html) that consume these lists.

a. Main components
- GitHub Repository (code + data at rest)
- GitHub Actions CI/CD
- Checks Workflow
- Regex Generation Workflow
- Update Workflow
- Node.js Processing Layer (download.js + package.json)
- Data Stores in repo: blacklist, whitelist, custom_filters, config, sources
- Output Artifacts: compiled regex files, updated JSON configs, update_history.txt
- External Hosting: GitHub Pages/raw.githubusercontent.com (serving list URLs)
- dnswarden Service (Frontend & Backend) consuming these lists
- End Users / Clients (DNS clients) querying dnswarden filters
b. Relationships and interactions
- Developers push commits to GitHub → triggers CI workflows
- Checks workflow validates JSON schemas, list integrity, hashes
- generateRegex workflow runs download.js → reads config + data folders → generates updated filter regex and JSON → commits/publishes artifacts
- update workflow updates update_history.txt and optionally increments version numbers
- Artifacts are published via raw.githubusercontent.com URLs
- dnswarden backend fetches these URLs for runtime filtering; dnswarden frontend lists them on customfilter/search pages
- End users configure their DNS clients to use dnswarden filters
c. Architectural patterns / design principles
- Infrastructure-as-code: GitHub Actions define build and publish pipelines
- Data-driven: directory structure holds data files, config drives processing
- Separation of concerns:
- Data (blacklist/whitelist directories)
- Processing (download.js / Node.js layer)
- CI/CD orchestration (GitHub Actions)
- Consumption (dnswarden UI & backend)
d. Key technologies / frameworks
- Node.js (download.js, package.json)
- JSON for configuration (config/*.json, edit_here_to_add_blocklists.json)
- GitHub Actions for CI/CD
- Raw file hosting via GitHub (GitHub Pages or raw.githubusercontent.com)
- DNS filtering service (dnswarden)

##### Essential elements
- Data flow directions and dependencies
- Related items (e.g., workflows together)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Actors
    Dev["Developer"]:::user
    EndUser["End Users / DNS Clients"]:::user

    %% GitHub Repository
    subgraph "GitHub Repository"
        Repo["Code & Data at Rest"]:::repo
        subgraph "Code & Scripts"
            DL["download.js"]:::code
            PKG["package.json"]:::code
        end
        subgraph "Config Files"
            CFG1["config/conn.json"]:::config
            CFG2["config/customfilter.json"]:::config
            CFG3["config/hashes.json"]:::config
            EDIT["edit_here_to_add_blocklists.json"]:::config
        end
        subgraph "Data Stores"
            BL["blacklist/"]:::data
            WL["whitelist/"]:::data
            CF["custom_filters/"]:::data
            SRC["sources/"]:::data
            TD["test_data/"]:::data
        end
        subgraph "CI/CD Pipeline Definition"
            WKS[".github/workflows/"]:::cicd
            subgraph "Workflows"
                CKS["Checks Workflow"]:::cicd
                RGX["Regex Generation Workflow"]:::cicd
                UPD["Update Workflow"]:::cicd
                DEL["Workflow Cleanup"]:::cicd
            end
        end
    end

    %% Processing Engine
    PE["Node.js Processing Engine"]:::process

    %% Output Artifacts
    subgraph "Artifacts"
        ART1["Compiled Regex & JSON"]:::artifact
        ART2["update_history.txt"]:::artifact
    end

    %% External Hosting
    EXT["raw.githubusercontent.com"]:::external

    %% dnswarden Service
    subgraph "dnswarden Service"
        BE["Backend"]:::service
        FE["Frontend"]:::service
    end

    %% Flows
    Dev -->|"push commits"| Repo
    Repo -->|"triggers workflows"| WKS
    CKS -->|"validates JSON & lists"| Repo
    RGX -->|"runs download.js"| PE
    UPD -->|"updates history"| ART2
    DEL -->|"cleans old workflows"| WKS

    PE -->|"reads data & config"| BL
    PE -->|"reads data & config"| WL
    PE -->|"reads data & config"| CF
    PE -->|"reads data & config"| CFG1
    PE -->|"reads data & config"| CFG2
    PE -->|"reads data & config"| CFG3
    PE -->|"reads data & config"| EDIT

    PE -->|"writes"| ART1
    PE -->|"writes"| ART2

    ART1 -->|"publishes lists"| EXT
    ART2 -->|"publishes history"| EXT

    BE -->|"fetches lists"| EXT
    FE -->|"displays lists"| EXT
    EndUser -->|"DNS queries"| BE

    %% Click Events
    click CKS "https://github.com/dnswarden/blocklist-staging/blob/main/.github/workflows/checks.yml"
    click RGX "https://github.com/dnswarden/blocklist-staging/blob/main/.github/workflows/generateRegex.yml"
    click UPD "https://github.com/dnswarden/blocklist-staging/blob/main/.github/workflows/update.yml"
    click DEL "https://github.com/dnswarden/blocklist-staging/blob/main/.github/workflows/delete_workflows.yml"
    click DL "https://github.com/dnswarden/blocklist-staging/blob/main/download.js"
    click PKG "https://github.com/dnswarden/blocklist-staging/blob/main/package.json"
    click BL "https://github.com/dnswarden/blocklist-staging/tree/main/blacklist/"
    click WL "https://github.com/dnswarden/blocklist-staging/tree/main/whitelist/"
    click CF "https://github.com/dnswarden/blocklist-staging/tree/main/custom_filters/"
    click CFG1 "https://github.com/dnswarden/blocklist-staging/blob/main/config/conn.json"
    click CFG2 "https://github.com/dnswarden/blocklist-staging/blob/main/config/customfilter.json"
    click CFG3 "https://github.com/dnswarden/blocklist-staging/blob/main/config/hashes.json"
    click EDIT "https://github.com/dnswarden/blocklist-staging/blob/main/edit_here_to_add_blocklists.json"
    click SRC "https://github.com/dnswarden/blocklist-staging/tree/main/sources/"
    click TD "https://github.com/dnswarden/blocklist-staging/tree/main/test_data/"
    click WKS "https://github.com/dnswarden/blocklist-staging/tree/main/.github/workflows/"

    %% Styles
    classDef code fill:#cce5ff,stroke:#339,stroke-width:1px
    classDef config fill:#e6ffcc,stroke:#396,stroke-width:1px
    classDef data fill:#e0ffe0,stroke:#2d862d,stroke-width:1px
    classDef cicd fill:#ffe5cc,stroke:#d99000,stroke-width:1px
    classDef repo fill:#f2f2f2,stroke:#999,stroke-width:1px
    classDef process fill:#d9d9ff,stroke:#6666cc,stroke-width:1px
    classDef artifact fill:#fff0b3,stroke:#cc9900,stroke-width:1px
    classDef external fill:#e6e6e6,stroke:#999,stroke-width:1px,stroke-dasharray: 5 5
    classDef service fill:#e0e0ff,stroke:#6666ff,stroke-width:1px
    classDef user fill:#ffccff,stroke:#cc0099,stroke-width:1px,stroke-dasharray: 2 2
```
