# shaharzelig/moneyman architecture

> Architecture diagram of the GitHub repository shaharzelig/moneyman, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shaharzelig/moneyman
- Repository: https://github.com/shaharzelig/moneyman

## Overview

##### Project Type and Purpose
- This is an ETL-style microservice (CLI/Docker) that orchestrates web scraping of Israeli bank/credit-card accounts, processes the transaction data, and exports it to one or more storage targets (Google Sheets, Azure Data Explorer, YNAB, Buxfer, JSON file, HTTP endpoint).
- A secondary “bot” component handles notifications and statistics reporting (via Telegram).

##### Main Architectural Components
a. Configuration Manager
- Reads all environment variables (ACCOUNTS_JSON, storage credentials, Telegram keys, domain-firewall rules, etc.).
b. Scraper Engine
- Leverages “israeli-bank-scrapers” (Puppeteer under the hood)
- Applies domain security rules (domainRules.ts, domains.ts)
- Runs in parallel up to MAX_PARALLEL_SCRAPERS
- Emits a unified list of Transaction objects
c. Data Processor
- Normalizes currency, timestamps, computes hashes (utils/currency.ts, transactionTableRow.ts)
d. Storage Adapter Layer (Plugin Pattern)
- JSON file (local)
- Google Sheets (src/bot/storage/sheets.ts)
- Azure Data Explorer (azure-data-explorer.ts)
- YNAB (ynab.ts)
- Buxfer (buxfer.ts)
- HTTP POST (web-post.ts)
e. Bot/Notifier
- Telegram notifier for run status & domain-access reports
- saveStats.ts collects run metadata (src/runnerMetadata.ts)
f. Runner / Orchestrator (src/index.ts)
- Entry point for CLI or Docker
- Coordinates config → scraper → processor → adapters → notifier

##### External Systems & Services
- Israeli bank websites (HTTPS endpoints)
- Telegram Bot API
- Google Sheets API
- Azure Data Explorer ingestion endpoint
- YNAB API
- Buxfer web interface/API
- Any HTTP endpoint for web-post export

##### Architectural Patterns & Principles
- Plugin pattern for storage adapters: new outputs can be added by implementing the Storage interface in src/bot/storage/*.
- Single responsibility: scraper vs. processor vs. notifier vs. adapters.
- Environment-driven configuration for secrets and feature flags (DOMAIN_TRACKING_ENABLED, FIREWALL_SETTINGS).
- Test coverage via Jest for each module (see jest.config.js).
- Containerized deployment (Dockerfile + docker-compose.yml + GitHub Actions).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        Config["Configuration Manager"]:::config
    end
    click Config "https://github.com/shaharzelig/moneyman/blob/main/src/config.ts"

    %% Runner / Orchestrator
    subgraph "Orchestrator"
        Runner["Runner / Orchestrator"]:::core
        Metadata["Run Metadata Collector"]:::core
    end
    click Runner "https://github.com/shaharzelig/moneyman/blob/main/src/index.ts"
    click Metadata "https://github.com/shaharzelig/moneyman/blob/main/src/runnerMetadata.ts"

    %% Core Pipeline
    subgraph "Core Pipeline"
        ScraperEngine["Scraper Engine\n(MAX_PARALLEL_SCRAPERS)"]:::core
        subgraph "Domain Security"
            Domains["Rule Definitions"]:::core
            Rules["Rule Application"]:::core
        end
        DataProcessor["Data Processor"]:::core
    end
    click ScraperEngine "https://github.com/shaharzelig/moneyman/blob/main/src/scraper/index.ts"
    click Browser "https://github.com/shaharzelig/moneyman/blob/main/src/scraper/browser.ts"
    click Scrape "https://github.com/shaharzelig/moneyman/blob/main/src/scraper/scrape.ts"
    click Cloudflare "https://github.com/shaharzelig/moneyman/blob/main/src/scraper/cloudflareSolver.ts"
    click Domains "https://github.com/shaharzelig/moneyman/blob/main/src/security/domains.ts"
    click Rules "https://github.com/shaharzelig/moneyman/blob/main/src/security/domainRules.ts"
    click Currency "https://github.com/shaharzelig/moneyman/blob/main/src/utils/currency.ts"
    click Row "https://github.com/shaharzelig/moneyman/blob/main/src/transactionTableRow.ts"
    click Utils "https://github.com/shaharzelig/moneyman/blob/main/src/utils/utils.ts"

    %% Storage Adapter Bus
    subgraph "Storage Adapter Bus"
        Registry["Adapter Registry"]:::adapter
        LocalJSON["Local JSON Adapter"]:::adapter
        Sheets["Google Sheets Adapter"]:::adapter
        ADE["Azure Data Explorer Adapter"]:::adapter
        YNAB["YNAB Adapter"]:::adapter
        Buxfer["Buxfer Adapter"]:::adapter
        WebPost["HTTP POST Adapter"]:::adapter
    end
    click Registry "https://github.com/shaharzelig/moneyman/blob/main/src/bot/storage/index.ts"
    click LocalJSON "https://github.com/shaharzelig/moneyman/blob/main/src/bot/storage/json.ts"
    click Sheets "https://github.com/shaharzelig/moneyman/blob/main/src/bot/storage/sheets.ts"
    click ADE "https://github.com/shaharzelig/moneyman/blob/main/src/bot/storage/azure-data-explorer.ts"
    click YNAB "https://github.com/shaharzelig/moneyman/blob/main/src/bot/storage/ynab.ts"
    click Buxfer "https://github.com/shaharzelig/moneyman/blob/main/src/bot/storage/buxfer.ts"
    click WebPost "https://github.com/shaharzelig/moneyman/blob/main/src/bot/storage/web-post.ts"

    %% Bot / Notifier
    subgraph "Bot / Notifier"
        Notifier["Notifier Service"]:::notifier
        Messages["Message Templates"]:::notifier
        SaveStats["Statistics Collector"]:::notifier
        BotEntry["Bot Entrypoint"]:::notifier
    end
    click Notifier "https://github.com/shaharzelig/moneyman/blob/main/src/bot/notifier.ts"
    click Messages "https://github.com/shaharzelig/moneyman/blob/main/src/bot/messages.ts"
    click SaveStats "https://github.com/shaharzelig/moneyman/blob/main/src/bot/saveStats.ts"
    click BotEntry "https://github.com/shaharzelig/moneyman/blob/main/src/bot/index.ts"

    %% External Systems
    subgraph "External Systems"
        Banks[(Bank Websites)]:::external
        FS[(Local File System)]:::external
        GS[(Google Sheets API)]:::external
        ADEAPI[(Azure Data Explorer)]:::external
        YNABAPI[(YNAB API)]:::external
        BuxAPI[(Buxfer API)]:::external
        HTTPEndpoint[(HTTP Endpoint)]:::external
        TelegramAPI[(Telegram Bot API)]:::external
    end

    %% Data Flows
    Config --> ScraperEngine
    Config --> Domains
    Config --> Runner
    Runner --> ScraperEngine
    Runner --> DataProcessor
    Runner --> Registry
    Runner --> Notifier
    ScraperEngine --> Domains
    Domains --> Banks
    ScraperEngine --> DataProcessor
    DataProcessor --> Registry
    Registry --> LocalJSON
    Registry --> Sheets
    Registry --> ADE
    Registry --> YNAB
    Registry --> Buxfer
    Registry --> WebPost
    LocalJSON --> FS
    Sheets --> GS
    ADE --> ADEAPI
    YNAB --> YNABAPI
    Buxfer --> BuxAPI
    WebPost --> HTTPEndpoint
    Notifier --> TelegramAPI

    %% Styles
    classDef config fill:#FFA500,stroke:#333,stroke-width:1px
    classDef core fill:#87CEFA,stroke:#333,stroke-width:1px
    classDef adapter fill:#98FB98,stroke:#333,stroke-width:1px
    classDef notifier fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
```
