# noblepayne/lnd-boost-scraper architecture

> Architecture diagram of the GitHub repository noblepayne/lnd-boost-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noblepayne/lnd-boost-scraper
- Repository: https://github.com/noblepayne/lnd-boost-scraper

## Overview

##### Main Components
- Upstream Integrations: Represent two distinct modules for fetching invoices from LND and Alby. In the file structure, these appear in the “src/boost_scraper/upstream” folder (alby.clj and lnd.clj). Label these as “LND API Connector” and “Alby API Connector.”
- Core Scraper & Aggregator: The main business logic lives in “src/boost_scraper/core.clj” and related files (like utils.clj and legacy.clj). This component orchestrates data fetching, processing, and coordination.
- Report Generation: The file “src/boost_scraper/reports.clj” handles the transformation of raw invoice/boost data into markdown formatted for show notes. This is your report generation module.
- Auxiliary Scripts & Resources: Include references to external scripts such as “scripts/compile_episode_list_from_markdown.clj” (for compiling markdown lists) and “resources/boost_report.js” (possibly used for UI interactivity or client-side report details). These can be depicted as supplementary tools that interact with or support the primary modules.

##### Relationships and Interactions
- Report Generation module to the Web Interface, indicating that users can trigger exports or view reports via the web server. Alternatively, if the report generation is an internal process, have the web interface pull from both the database and the report generator.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Clearly distinguish between data ingestion, storage, processing, and presentation.
- Modular Design: Highlight that components are arranged in discrete namespaces (e.g., “upstream”, “core”, “db”, “reports”, “web”), which makes it easier to extend or modify individual pieces.
- Integration Points: Identify external dependencies (LND, Alby, additional scripts and the Nix build/deployment system) to clarify how the internal components interact with outside systems.

##### Relevant Technologies
- Clearly label that the system is built in Clojure.
- Data Persistence element with “Datalevin” to stress the chosen database engine.
- Deployment/build configuration using Nix (as seen with flake.nix, module.nix, and deps.edn) to show how dependencies and environments are managed.
- If applicable, note that “devenv” and clj-nix are used to streamline development.

## Diagram (Mermaid)

```mermaid
flowchart TD
    ENV["Environment/Nix Config"]:::env

    subgraph "Data Ingestion & Upstream"
        LND["LND API Connector"]:::api
        ALBY["Alby API Connector"]:::api
    end

    subgraph "Core Processing"
        CORE["Core Logic & Aggregation"]:::core
        UTILS["Utils"]:::core
        LEGACY["Legacy"]:::core
    end

    subgraph "Data Storage"
        DB["Datalevin Local Database"]:::db
    end

    subgraph "Reporting & Presentation"
        REPORT["Markdown Report Generator"]:::report
        WEB["Web/Export Interface"]:::web
    end

    subgraph "Auxiliary Tools"
        SCRIPT["Compile Episode List Script"]:::aux
        JS["Boost Report JS Resource"]:::aux
    end

    ENV -->|"provides_config"| CORE
    LND -->|"data_fetch"| CORE
    ALBY -->|"data_fetch"| CORE
    CORE -->|"store_invoice"| DB
    CORE -->|"generate_markdown"| REPORT
    REPORT -->|"export_data"| WEB
    SCRIPT -.->|"compiles_markdown"| REPORT
    JS -.->|"enhances_ui"| WEB
    CORE --- UTILS
    CORE --- LEGACY

    click LND "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/src/boost_scraper/upstream/lnd.clj"
    click ALBY "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/src/boost_scraper/upstream/alby.clj"
    click CORE "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/src/boost_scraper/core.clj"
    click UTILS "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/src/boost_scraper/utils.clj"
    click LEGACY "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/src/boost_scraper/legacy.clj"
    click DB "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/src/boost_scraper/db.clj"
    click REPORT "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/src/boost_scraper/reports.clj"
    click WEB "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/src/boost_scraper/web.clj"
    click SCRIPT "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/scripts/compile_episode_list_from_markdown.clj"
    click JS "https://github.com/noblepayne/lnd-boost-scraper/blob/v2/resources/boost_report.js"

    classDef api fill:#FFEE88,stroke:#CC9900,stroke-width:2px,color:#000;
    classDef core fill:#99CCFF,stroke:#336699,stroke-width:2px,color:#000;
    classDef db fill:#FFCCCC,stroke:#CC3333,stroke-width:2px,color:#000;
    classDef report fill:#CCFFCC,stroke:#33CC33,stroke-width:2px,color:#000;
    classDef web fill:#CCCCFF,stroke:#3333CC,stroke-width:2px,color:#000;
    classDef aux fill:#F0E68C,stroke:#BDB76B,stroke-width:2px,color:#000;
    classDef env fill:#D3D3D3,stroke:#696969,stroke-width:2px,color:#000;
```
