How borestad/blocklist-abuseipdb works
This project is a data aggregation and processing tool that creates and maintains blocklists of offenders (IPv4 and IPv6) using data from AbuseIPDB. The repository is built around scheduled (cron) tasks, file‐based storage of lists (grouped by time windows and confidence levels), and additional statistical “hall of shame” reports. The whole process is orchestrated with Deno scripts and a simple CI/CD integration via GitHub Actions.
Components Identification
a. Cron Scheduler Component:
• .cron folder as the scheduler layer.
• Jobs (e.g. abuseipdb jobs with subfolders for different update tasks) and corresponding scripts (in .cron/scripts such as “ip2ipinfo.ts”) that run on a schedule.
b. Data Processing and Aggregation Module:
• Deno environment (deno.json, deno.lock) and associated scripts that fetch data from AbuseIPDB and enrich it with metadata from ipinfo.io (using the ASN database).
• Transformation process that results in daily or time-based blocklist files (such as abuseipdb-s100-1d, 3d, etc.).
c. Data Storage and Aggregated Outputs:
• “db” directory which stores raw or processed IP list files segmented by date.
• “stats” directory that contains additional aggregated statistics (e.g. stats/blocks, stats/hallofshame, and stats/ips).
d. CI/CD Pipeline:
• GitHub workflows (from .github/workflows) that run automated tests or validations and possibly trigger the cron jobs.
e. Configuration/Utility Components:
• Utility build/deploy configurations such as the Justfile (in .cron/Justfile) and package.json which manage dependencies.
Relationships and Data-Flow
b. Data Processing module to the Data Storage “db” and “stats” directories to indicate that processed results are saved as file lists.
d. CI/CD pipeline (GitHub workflows) as an orchestrator that monitors code and, when changes occur, triggers the build and possibly testing of scheduled scripts.
Architectural Patterns and Design Principles
a. Batch processing pattern: scheduled scripts executing periodic fetch, process, and store tasks.
b. Use of file‐based data storage where each day’s data is stored in its own dated folder under “db.”
c. Separation of concerns: scheduling, processing/enrichment, and storage/aggregation are handled in distinct sets of files and directories.
Technologies and Frameworks
a. Deno is the runtime (as indicated by deno.json, deno.lock, and TypeScript files such as ip2ipinfo.ts).
b. GitHub Actions (configured in .github/workflows) provides the CI/CD framework.
c. External APIs from AbuseIPDB and ipinfo.io are an integral part of the data enrichment.