How shaharzelig/moneyman works
Project Type and Purpose
• This is an ETL-style microservice (CLI/Docker) that orchestrates web scraping of Israeli bank/credit-card accounts, processes the transaction data, and exports it to one or more storage targets (Google Sheets, Azure Data Explorer, YNAB, Buxfer, JSON file, HTTP endpoint).
• A secondary “bot” component handles notifications and statistics reporting (via Telegram).
Main Architectural Components
a. Configuration Manager
• Reads all environment variables (ACCOUNTS_JSON, storage credentials, Telegram keys, domain-firewall rules, etc.).
b. Scraper Engine
• Leverages “israeli-bank-scrapers” (Puppeteer under the hood)
• Applies domain security rules (domainRules.ts, domains.ts)
• Runs in parallel up to MAX_PARALLEL_SCRAPERS
• Emits a unified list of Transaction objects
c. Data Processor
• Normalizes currency, timestamps, computes hashes (utils/currency.ts, transactionTableRow.ts)
d. Storage Adapter Layer (Plugin Pattern)
• JSON file (local)
• Google Sheets (src/bot/storage/sheets.ts)
• Azure Data Explorer (azure-data-explorer.ts)
• YNAB (ynab.ts)
• Buxfer (buxfer.ts)
• HTTP POST (web-post.ts)
e. Bot/Notifier
• Telegram notifier for run status & domain-access reports
• saveStats.ts collects run metadata (src/runnerMetadata.ts)
f. Runner / Orchestrator (src/index.ts)
• Entry point for CLI or Docker
• Coordinates config → scraper → processor → adapters → notifier
External Systems & Services
• Israeli bank websites (HTTPS endpoints)
• Telegram Bot API
• Google Sheets API
• Azure Data Explorer ingestion endpoint
• YNAB API
• Buxfer web interface/API
• Any HTTP endpoint for web-post export
Architectural Patterns & Principles
• Plugin pattern for storage adapters: new outputs can be added by implementing the Storage interface in src/bot/storage/*.
• Single responsibility: scraper vs. processor vs. notifier vs. adapters.
• Environment-driven configuration for secrets and feature flags (DOMAIN_TRACKING_ENABLED, FIREWALL_SETTINGS).
• Test coverage via Jest for each module (see jest.config.js).
• Containerized deployment (Dockerfile + docker-compose.yml + GitHub Actions).