How adamsarek/btc-address-lookup works
Project type and purpose
• This is a two-module system:
a. BTCAbuseCrawler (Python) – a multi-threaded data-collection pipeline that crawls public abuse-report websites, parses and normalizes the data, writes raw files and populates a PostgreSQL database.
b. BTCAbuseSearch (Node.js/Express + EJS) – a full-stack web service and REST API that authenticates users, issues tokens, queries the same PostgreSQL database, caches results, and serves both HTML views and JSON.
High-level file structure analysis
• Top level:
• btc_abuse_crawler/
– console (entrypoint CLI)
– crawler.py (crawl scheduler)
– request/response (HTTP layer + parsers)
– mapper (normalize HTML/JSON into domain models)
– file (raw file I/O)
– database (schema init + adapters)
– main.py (orchestrates setup, crawler run)
• btc_abuse_search/
– main.js (Express server entrypoint)
– database/*.js (DB connection & queries)
– public/src (static frontend assets: CSS, images)
– views (EJS templates + partials)
– routes & controllers implicit in main.js
Key insights from README
• Both tools share the same PostgreSQL schema.
• Crawler respects robots.txt, multi-threads sources, writes raw data to disk and structured tables.
• Search service offers role-based REST API endpoints, token generation, data caching, and server-rendered web pages (MVC pattern).
• BTCAbuseCrawler service (Python):
– Scheduler/Controller
– HTTP Requester & Robots.txt module
– Parsers (HTML, JSON)
– Mapper/Normalizer
– File Store (raw data directory)
– Database Adapter (DB init / schema management)
• BTCAbuseSearch service (Node.js/Express):
– API Layer (token auth, rate limiting, caching)
– MVC: Controllers → Views (EJS) + Public static assets
– Database Adapter (queries, pooling)
• Client types:
– Web Browser (HTML pages)
– API Consumer (JSON clients)
• Crawler → External Sources (polling/crawling)
• Crawler → File Store (write raw)
• Crawler → PostgreSQL (INSERT/UPDATE)
• Web/API clients → Search Service (HTTP requests)
• Search Service → PostgreSQL (SELECT)
• Search Service → Cache (in-memory or external, if used)
• Search Service → Web Browser (rendered HTML + static assets)
c. Architectural patterns & principles:
• Separation of concerns: data ingestion vs. data serving
• Pipeline pattern in crawler: fetch → parse → map → persist
• MVC pattern in search: Models (DB), Views (EJS), Controllers (Express routes)
• Role-based access control & token auth in search API
• Multi-threading for crawler concurrency
d. Key technologies & frameworks:
• Python 3.11 – threading, requests, JSON/HTML parsing
• PostgreSQL 15.2 – relational storage, schema defined in database.py
• Node.js 18.15 LTS – Express.js, EJS templating
• npm modules – caching, authentication, database drivers