How dnswarden/blocklist-staging works
Project type and purpose
• This is an open-source “staging blacklist/whitelist/customlist” repository for the dnswarden service.
• It contains raw data (blacklists, whitelists, custom filters), configuration files, a Node.js script (download.js), and GitHub Actions workflows to validate, generate, and update blocklists for dnswarden.
File structure
• Top-level artifacts:
• .github/workflows – CI workflows: checks, generateRegex, update
• config – JSON files (conn.json, customfilter.json, hashes.json)
• blacklist, whitelist, custom_filters – raw domain lists and custom filter blobs
• sources – metadata files (.md) describing each source list
• download.js – main Node.js script that pulls config, fetches or reads lists and produces outputs
• edit_here_to_add_blocklists.json – config for default lists used by dnswarden UI
• package.json – Node.js dependencies and scripts
• test_data – example inputs/outputs for local testing
• GitHub Actions orchestrate: linting/validation (.github/workflows/checks.yml), building regex files (generateRegex.yml), updating histories (update.yml).
Insights from README
• Describes how to add or remove blocklists via edit_here_to_add_blocklists.json
• Explains the schema of list entries (name, category, url[], filterType[], value, enabled)
• Points to dnswarden’s UI endpoints (customfilter.html, search.html) that consume these lists.
a. Main components
• GitHub Repository (code + data at rest)
• GitHub Actions CI/CD
• Checks Workflow
• Regex Generation Workflow
• Update Workflow
• Node.js Processing Layer (download.js + package.json)
• Data Stores in repo: blacklist, whitelist, custom_filters, config, sources
• Output Artifacts: compiled regex files, updated JSON configs, update_history.txt
• External Hosting: GitHub Pages/raw.githubusercontent.com (serving list URLs)
• dnswarden Service (Frontend & Backend) consuming these lists
• End Users / Clients (DNS clients) querying dnswarden filters
b. Relationships and interactions
• Developers push commits to GitHub → triggers CI workflows
• Checks workflow validates JSON schemas, list integrity, hashes
• generateRegex workflow runs download.js → reads config + data folders → generates updated filter regex and JSON → commits/publishes artifacts
• update workflow updates update_history.txt and optionally increments version numbers
• Artifacts are published via raw.githubusercontent.com URLs
• dnswarden backend fetches these URLs for runtime filtering; dnswarden frontend lists them on customfilter/search pages
• End users configure their DNS clients to use dnswarden filters
c. Architectural patterns / design principles
• Infrastructure-as-code: GitHub Actions define build and publish pipelines
• Data-driven: directory structure holds data files, config drives processing
• Separation of concerns:
• Data (blacklist/whitelist directories)
• Processing (download.js / Node.js layer)
• CI/CD orchestration (GitHub Actions)
• Consumption (dnswarden UI & backend)
d. Key technologies / frameworks
• Node.js (download.js, package.json)
• JSON for configuration (config/*.json, edit_here_to_add_blocklists.json)
• GitHub Actions for CI/CD
• Raw file hosting via GitHub (GitHub Pages or raw.githubusercontent.com)
• DNS filtering service (dnswarden)
Essential elements
• Data flow directions and dependencies
• Related items (e.g., workflows together)