# adamsarek/btc-address-lookup architecture

> Architecture diagram of the GitHub repository adamsarek/btc-address-lookup, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/adamsarek/btc-address-lookup
- Repository: https://github.com/adamsarek/btc-address-lookup

## Overview

##### Project type and purpose
- This is a two-module system:
a. BTCAbuseCrawler (Python) – a multi-threaded data-collection pipeline that crawls public abuse-report websites, parses and normalizes the data, writes raw files and populates a PostgreSQL database.
b. BTCAbuseSearch (Node.js/Express + EJS) – a full-stack web service and REST API that authenticates users, issues tokens, queries the same PostgreSQL database, caches results, and serves both HTML views and JSON.

##### High-level file structure analysis
- Top level:
- btc_abuse_crawler/
– console (entrypoint CLI)
– crawler.py (crawl scheduler)
– request/response (HTTP layer + parsers)
– mapper (normalize HTML/JSON into domain models)
– file (raw file I/O)
– database (schema init + adapters)
– main.py (orchestrates setup, crawler run)
- btc_abuse_search/
– main.js (Express server entrypoint)
– database/*.js (DB connection & queries)
– public/src (static frontend assets: CSS, images)
– views (EJS templates + partials)
– routes & controllers implicit in main.js

##### Key insights from README
- Both tools share the same PostgreSQL schema.
- Crawler respects robots.txt, multi-threads sources, writes raw data to disk and structured tables.
- Search service offers role-based REST API endpoints, token generation, data caching, and server-rendered web pages (MVC pattern).

- BTCAbuseCrawler service (Python):
– Scheduler/Controller
– HTTP Requester & Robots.txt module
– Parsers (HTML, JSON)
– Mapper/Normalizer
– File Store (raw data directory)
– Database Adapter (DB init / schema management)
- BTCAbuseSearch service (Node.js/Express):
– API Layer (token auth, rate limiting, caching)
– MVC: Controllers → Views (EJS) + Public static assets
– Database Adapter (queries, pooling)
- Client types:
– Web Browser (HTML pages)
– API Consumer (JSON clients)

- Crawler → External Sources (polling/crawling)
- Crawler → File Store (write raw)
- Crawler → PostgreSQL (INSERT/UPDATE)
- Web/API clients → Search Service (HTTP requests)
- Search Service → PostgreSQL (SELECT)
- Search Service → Cache (in-memory or external, if used)
- Search Service → Web Browser (rendered HTML + static assets)

c. Architectural patterns & principles:
- Separation of concerns: data ingestion vs. data serving
- Pipeline pattern in crawler: fetch → parse → map → persist
- MVC pattern in search: Models (DB), Views (EJS), Controllers (Express routes)
- Role-based access control & token auth in search API
- Multi-threading for crawler concurrency

d. Key technologies & frameworks:
- Python 3.11 – threading, requests, JSON/HTML parsing
- PostgreSQL 15.2 – relational storage, schema defined in database.py
- Node.js 18.15 LTS – Express.js, EJS templating
- npm modules – caching, authentication, database drivers

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources
    ExternalWebsites(("External Sources\n(Websites)")):::external

    %% Data Pipeline (BTCAbuseCrawler)
    subgraph "Data Pipeline (BTCAbuseCrawler)"
        CLI["CLI Entrypoint"]:::crawler
        Scheduler["Crawler Scheduler"]:::crawler
        Orchestrator["Orchestrator (main.py)"]:::crawler
        Requester["HTTP Requester & Robots.txt"]:::crawler
        HTMLParser["HTML Response Parser"]:::crawler
        JSONParser["JSON Response Parser"]:::crawler
        Mapper["Mapper / Normalizer"]:::crawler
        FileStore[/Raw File Store/]:::file
        DBAdapterCrawler["DB Adapter (Schema Init & Adapters)"]:::dbadapter
    end

    %% Persistence
    subgraph "Persistence"
        DB[(PostgreSQL Database)]:::database
    end

    %% Web Service (BTCAbuseSearch)
    subgraph "Web Service (BTCAbuseSearch)"
        Express["Express Server"]:::search
        APILayer["API Layer\n(Token Auth, Rate Limiting, Caching)"]:::search
        Controllers["Controllers"]:::search
        Views[>EJS Views/]:::search
        StaticAssets[[/Static Assets/]]:::search
        DBAdapterSearch["DB Adapter (Queries & Pooling)"]:::dbadapter
    end

    %% Clients
    subgraph "Clients"
        Browser(("Web Browser")):::external
        APIClient(("API Consumer")):::external
    end

    %% Data Pipeline Flow
    CLI --> Scheduler
    Scheduler --> Orchestrator
    Orchestrator --> Requester
    ExternalWebsites --> Requester
    Requester --> HTMLParser
    Requester --> JSONParser
    HTMLParser --> Mapper
    JSONParser --> Mapper
    Mapper --> FileStore
    Mapper --> DBAdapterCrawler
    DBAdapterCrawler --> DB

    %% Web Service Flow
    Browser --> Express
    APIClient --> Express
    Express --> APILayer
    APILayer --> Controllers
    APILayer --> APIClient
    Controllers --> Views
    Controllers --> StaticAssets
    Controllers --> DBAdapterSearch
    DBAdapterSearch --> DB
    Views --> Browser
    StaticAssets --> Browser

    %% Click Events
    click CLI "https://github.com/adamsarek/btc-address-lookup/blob/main/btc_abuse_crawler/console/console.py"
    click Scheduler "https://github.com/adamsarek/btc-address-lookup/blob/main/btc_abuse_crawler/crawler.py"
    click Orchestrator "https://github.com/adamsarek/btc-address-lookup/blob/main/btc_abuse_crawler/main.py"
    click Requester "https://github.com/adamsarek/btc-address-lookup/blob/main/btc_abuse_crawler/request/request.py"
    click HTMLParser "https://github.com/adamsarek/btc-address-lookup/blob/main/btc_abuse_crawler/response/html_response.py"
    click JSONParser "https://github.com/adamsarek/btc-address-lookup/blob/main/btc_abuse_crawler/response/response.py"
    click Mapper "https://github.com/adamsarek/btc-address-lookup/blob/main/btc_abuse_crawler/mapper/mapper.py"
    click FileStore "https://github.com/adamsarek/btc-address-lookup/tree/main/btc_abuse_crawler/file/"
    click DBAdapterCrawler "https://github.com/adamsarek/btc-address-lookup/tree/main/btc_abuse_crawler/database/"
    click Express "https://github.com/adamsarek/btc-address-lookup/blob/main/btc_abuse_search/main.js"
    click DBAdapterSearch "https://github.com/adamsarek/btc-address-lookup/tree/main/btc_abuse_search/database/"
    click StaticAssets "https://github.com/adamsarek/btc-address-lookup/tree/main/btc_abuse_search/public/src/"
    click Views "https://github.com/adamsarek/btc-address-lookup/tree/main/btc_abuse_search/views/"

    %% Styles
    classDef crawler fill:#e0f7fa,stroke:#006064,color:#000
    classDef search fill:#ffe0b2,stroke:#e65100,color:#000
    classDef database fill:#c8e6c9,stroke:#2e7d32,color:#000
    classDef external fill:#eeeeee,stroke:#757575,color:#000
    classDef file fill:#e1f5fe,stroke:#0277bd,color:#000
    classDef dbadapter fill:#bbdefb,stroke:#0d47a1,color:#000
```
