How noblepayne/lnd-boost-scraper works
Main Components
• Upstream Integrations: Represent two distinct modules for fetching invoices from LND and Alby. In the file structure, these appear in the “src/boost_scraper/upstream” folder (alby.clj and lnd.clj). Label these as “LND API Connector” and “Alby API Connector.”
• Core Scraper & Aggregator: The main business logic lives in “src/boost_scraper/core.clj” and related files (like utils.clj and legacy.clj). This component orchestrates data fetching, processing, and coordination.
• Report Generation: The file “src/boost_scraper/reports.clj” handles the transformation of raw invoice/boost data into markdown formatted for show notes. This is your report generation module.
• Auxiliary Scripts & Resources: Include references to external scripts such as “scripts/compile_episode_list_from_markdown.clj” (for compiling markdown lists) and “resources/boost_report.js” (possibly used for UI interactivity or client-side report details). These can be depicted as supplementary tools that interact with or support the primary modules.
Relationships and Interactions
• Report Generation module to the Web Interface, indicating that users can trigger exports or view reports via the web server. Alternatively, if the report generation is an internal process, have the web interface pull from both the database and the report generator.
Architectural Patterns and Design Principles
• Separation of Concerns: Clearly distinguish between data ingestion, storage, processing, and presentation.
• Modular Design: Highlight that components are arranged in discrete namespaces (e.g., “upstream”, “core”, “db”, “reports”, “web”), which makes it easier to extend or modify individual pieces.
• Integration Points: Identify external dependencies (LND, Alby, additional scripts and the Nix build/deployment system) to clarify how the internal components interact with outside systems.
Relevant Technologies
• Clearly label that the system is built in Clojure.
• Data Persistence element with “Datalevin” to stress the chosen database engine.
• Deployment/build configuration using Nix (as seen with flake.nix, module.nix, and deps.edn) to show how dependencies and environments are managed.
• If applicable, note that “devenv” and clj-nix are used to streamline development.