# mayilian/waycore-bank-scraper architecture

> Architecture diagram of the GitHub repository mayilian/waycore-bank-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mayilian/waycore-bank-scraper
- Repository: https://github.com/mayilian/waycore-bank-scraper

## Overview

1. **Overall shape**
   - WayCore is a two-service system built around a shared Postgres database: a **FastAPI API** for tenant-facing actions and a **worker** for browser automation.
   - The services communicate through **Restate**, which provides durable workflows, checkpointing, pause/resume, and replay after crashes.

2. **Request and control plane**
   - The **API service** exposes connection/sync/job/account/transaction endpoints under `src/api/routes/`.
   - It is intentionally thin: routes validate input, authenticate via API key, then delegate to shared business logic in `src/core/operations.py` and tenant-scoped DB helpers in `src/db/queries.py`.
   - This is the boundary for clients and integrations; the API never runs Playwright itself.

3. **Durable sync workflow**
   - The sync pipeline lives in `src/worker/workflow.py` and is the core orchestration boundary.
   - The workflow is structured as **login → extract_all → finalise**, with Restate journaling each step so a crash resumes from the last checkpoint rather than starting over.
   - OTP handling is designed as a suspend/resume point: the workflow can wait without holding browser resources.

4. **Browser automation plane**
   - The **worker service** runs Playwright-based browser sessions, stealth helpers, screenshots, and URL normalization logic from `src/core/stealth.py`, `src/core/screenshots.py`, and `src/core/urls.py`.
   - The login browser captures authenticated state; the extraction browser reuses that session to visit accounts and collect balances and transactions.
   - The worker is the only place where bank portals are touched directly.

5. **Bank adapter boundary**
   - Bank-specific behavior is isolated under `src/adapters/`.
   - `BankAdapter` is the abstraction, with a deterministic Heritage Trust adapter and a generic fallback for unknown banks.
   - This is the main extensibility seam: adding a bank means adding an adapter, not changing workflow or API layers.

6. **Extraction strategy**
   - Extraction is tiered: deterministic DOM parsing first, then LLM text fallback, then LLM vision fallback if needed.
   - The LLM machinery in `src/agent/llm.py` and `src/agent/extractor.py` is only invoked when selectors or parsing fail, which keeps the happy path fast and cheap.
   - The Heritage demo adapter mostly uses pure parsing functions in `src/adapters/heritage_parsers.py`, so browser automation and data extraction remain separated.

7. **Persistence model**
   - `src/db/models.py` defines the relational model around tenants, API keys, bank connections, sync jobs, account sync results, accounts, transactions, and balance history.
   - Writes are designed to be idempotent: transactions use conflict-avoiding inserts, balances are append-only, and sync results track partial success.
   - Migrations under `alembic/` represent the schema evolution for sync tracking, URL normalization, deduplication, and API keys.

8. **Tenant isolation and auth**
   - Tenant access is enforced in `src/api/auth.py` using API keys hashed with SHA-256 and checked with timing-safe comparison.
   - All reads and writes are scoped through tenant-aware query helpers in `src/db/queries.py`; the API layer does not expose raw cross-tenant model access.
   - This is a deliberate application-level boundary for multi-tenant safety.

9. **Credential and secrets handling**
   - Bank credentials are encrypted at rest with MultiFernet in `src/core/crypto.py`.
   - Decryption happens only inside worker memory, and key rotation is supported without re-encrypting the database.
   - External secrets also include LLM provider keys or AWS IAM-based Bedrock access, depending on deployment configuration.

10. **Concurrency and rate protection**
   - `src/worker/concurrency.py` enforces both global worker limits and per-bank semaphore limits.
   - This protects the worker from browser overload and reduces the risk of bank-side throttling or IP blocking.
   - The architecture assumes workflow queueing is preferable to uncontrolled parallel browser launches.

11. **CLI and shared application logic**
   - `cli.py` provides operational commands for sync execution, OTP submission, and data inspection.
   - The CLI and API both reuse the same business logic layer in `src/core/operations.py`, so the control plane is shared even though the entrypoints differ.
   - In local development, the CLI can talk directly to Postgres, while the API path goes through auth and tenant scoping.

12. **Runtime and deployment**
   - The project is packaged as a single Docker image but deployed as two ECS services with different resource profiles: API is lightweight, worker is memory-heavy.
   - AWS CDK under `deploy/cdk/` splits infrastructure into **Foundation** and **App** stacks, with ECR, VPC, RDS Postgres, S3 screenshots, Secrets Manager, and Restate as supporting infrastructure.
   - Restate and Cloud Map provide private service-to-service routing inside the VPC.

13. **Observability and operational concerns**
   - `src/core/logging.py` and `src/core/metrics.py` provide structured logs and CloudWatch EMF metrics.
   - Health checks are exposed through the API, while screenshots and workflow traces help diagnose extraction failures.
   - The system is built to be crash-tolerant rather than perfectly synchronous, so observability is tied to workflow steps and DB state.

14. **Testing and validation**
   - Unit tests cover adapters, crypto, extraction, models, and URL handling; integration tests cover API behavior and failure modes against Postgres.
   - This mirrors the architecture: pure logic is testable without browsers, while end-to-end behavior depends on workflow, DB, and browser execution together.

Main entry points: [`src/worker/workflow.py`](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/workflow.py) (Workflow), [`src/db/queries.py`](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/db/queries.py) (Queries), [`deploy/cdk/waycore_stack.py`](https://github.com/mayilian/waycore-bank-scraper/blob/main/deploy/cdk/waycore_stack.py) (CDK), [`src/api/routes`](https://github.com/mayilian/waycore-bank-scraper/tree/main/src/api/routes) (Routes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_api["API & control"]
  node_node_api_app["API app<br/>FastAPI service<br/>[app.py]"]
  node_node_api_routes["Routes<br/>HTTP handlers"]
  node_node_api_auth["Auth<br/>[auth.py]"]
  node_node_operations["Operations<br/>shared business logic<br/>[operations.py]"]
  node_node_cli["CLI<br/>ops entrypoint<br/>[cli.py]"]
end

subgraph group_group_worker["Worker & browser"]
  node_node_workflow["Workflow<br/>durable sync orchestration<br/>[workflow.py]"]
  node_node_steps["Steps<br/>workflow actions<br/>[steps.py]"]
  node_node_concurrency["Concurrency<br/>worker limits<br/>[concurrency.py]"]
  node_node_worker_app["Worker app<br/>worker service<br/>[app.py]"]
  node_node_stealth["Stealth<br/>browser helpers<br/>[stealth.py]"]
  node_node_screenshots["Screenshots<br/>diagnostics<br/>[screenshots.py]"]
  node_node_adapters["Adapters<br/>bank boundary"]
  node_node_heritage["Heritage adapter<br/>deterministic adapter"]
  node_node_generic["Generic adapter<br/>fallback adapter"]
  node_node_extract["Extractor<br/>tiered extraction<br/>[extractor.py]"]
  node_node_llm["LLM<br/>model access<br/>[llm.py]"]
end

subgraph group_group_data["Data & persistence"]
  node_node_db_models[("Models<br/>relational schema<br/>[models.py]")]
  node_node_db_queries["Queries<br/>tenant-scoped db access<br/>[queries.py]"]
  node_node_crypto["Crypto<br/>secrets handling<br/>[crypto.py]"]
  node_node_db_session["DB session<br/>postgres access<br/>[session.py]"]
end

subgraph group_group_platform["Platform & deploy"]
  node_node_metrics["Metrics<br/>observability<br/>[metrics.py]"]
  node_node_logging["Logging<br/>observability<br/>[logging.py]"]
  node_node_deploy["CDK<br/>infrastructure<br/>[waycore_stack.py]"]
end

node_node_api_app -->|"hosts"| node_node_api_routes
node_node_api_routes -->|"authenticates"| node_node_api_auth
node_node_api_routes -->|"delegates"| node_node_operations
node_node_operations -->|"reads/writes"| node_node_db_queries
node_node_db_queries -->|"maps"| node_node_db_models
node_node_db_queries -->|"uses"| node_node_db_session
node_node_api_routes -->|"scoped access"| node_node_db_queries
node_node_cli -->|"reuses"| node_node_operations
node_node_cli -->|"direct db"| node_node_db_session
node_node_worker_app -->|"runs"| node_node_workflow
node_node_workflow -->|"orchestrates"| node_node_steps
node_node_workflow -->|"limits"| node_node_concurrency
node_node_workflow -->|"adapts"| node_node_adapters
node_node_steps -->|"uses"| node_node_stealth
node_node_steps -->|"captures"| node_node_screenshots
node_node_adapters -->|"includes"| node_node_heritage
node_node_adapters -->|"includes"| node_node_generic
node_node_heritage -->|"parses via"| node_node_extract
node_node_extract -->|"falls back to"| node_node_llm
node_node_workflow -->|"persists state"| node_node_db_queries
node_node_crypto -->|"decrypts creds"| node_node_workflow
node_node_api_auth -->|"validates keys"| node_node_db_models
node_node_logging -.->|"instrumentation"| node_node_api_app
node_node_metrics -.->|"instrumentation"| node_node_worker_app
node_node_deploy -->|"deploys"| node_node_api_app
node_node_deploy -->|"deploys"| node_node_worker_app
node_node_deploy -->|"provisions"| node_node_db_session
node_node_deploy -->|"integrates"| node_node_workflow

click node_node_api_app "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/api/app.py"
click node_node_api_routes "https://github.com/mayilian/waycore-bank-scraper/tree/main/src/api/routes"
click node_node_api_auth "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/api/auth.py"
click node_node_operations "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/operations.py"
click node_node_cli "https://github.com/mayilian/waycore-bank-scraper/blob/main/cli.py"
click node_node_workflow "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/workflow.py"
click node_node_steps "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/steps.py"
click node_node_concurrency "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/concurrency.py"
click node_node_worker_app "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/app.py"
click node_node_stealth "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/stealth.py"
click node_node_screenshots "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/screenshots.py"
click node_node_adapters "https://github.com/mayilian/waycore-bank-scraper/tree/main/src/adapters"
click node_node_heritage "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/adapters/heritage_bank_adapter.py"
click node_node_generic "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/adapters/generic_bank_adapter.py"
click node_node_extract "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/agent/extractor.py"
click node_node_llm "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/agent/llm.py"
click node_node_db_models "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/db/models.py"
click node_node_db_queries "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/db/queries.py"
click node_node_crypto "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/crypto.py"
click node_node_db_session "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/db/session.py"
click node_node_metrics "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/metrics.py"
click node_node_logging "https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/logging.py"
click node_node_deploy "https://github.com/mayilian/waycore-bank-scraper/blob/main/deploy/cdk/waycore_stack.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_api_app,node_node_api_routes,node_node_api_auth,node_node_operations,node_node_cli toneBlue
class node_node_workflow,node_node_steps,node_node_concurrency,node_node_worker_app,node_node_stealth,node_node_screenshots,node_node_adapters,node_node_heritage,node_node_generic,node_node_extract,node_node_llm toneAmber
class node_node_db_models,node_node_db_queries,node_node_crypto,node_node_db_session toneMint
class node_node_metrics,node_node_logging,node_node_deploy toneRose
```

## Components

### API & control

Tenant-facing request handling and shared business operations.

- [API app](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/api/app.py): FastAPI service · Main HTTP entrypoint for tenant-facing actions and health checks. · `src/api/app.py`
- [Routes](https://github.com/mayilian/waycore-bank-scraper/tree/main/src/api/routes): HTTP handlers · Connection, sync, account, transaction, and health endpoints. · `src/api/routes`
- [Auth](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/api/auth.py): API key auth · Tenant authentication and access control based on hashed API keys. · `src/api/auth.py`
- [Operations](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/operations.py): shared business logic · Core application logic reused by API and CLI entrypoints. · `src/core/operations.py`
- [CLI](https://github.com/mayilian/waycore-bank-scraper/blob/main/cli.py): ops entrypoint · Operational commands for sync execution, OTP submission, and inspection. · `cli.py`

### Worker & browser

Durable sync workflow and Playwright-based portal automation.

- [Workflow](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/workflow.py): durable sync orchestration · Restate-backed login → extract → finalise pipeline with checkpointing and resume. · `src/worker/workflow.py`
- [Steps](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/steps.py): workflow actions · Browser and sync step implementations used by the durable workflow. · `src/worker/steps.py`
- [Concurrency](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/concurrency.py): worker limits · Global and per-bank semaphore controls to prevent overload and throttling. · `src/worker/concurrency.py`
- [Worker app](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/worker/app.py): worker service · Worker process entrypoint that runs automation and workflow execution. · `src/worker/app.py`
- [Stealth](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/stealth.py): browser helpers · Playwright stealth and browser-behavior helpers for portal access. · `src/core/stealth.py`
- [Screenshots](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/screenshots.py): diagnostics · Screenshot capture used for debugging and workflow failure analysis. · `src/core/screenshots.py`
- [Adapters](https://github.com/mayilian/waycore-bank-scraper/tree/main/src/adapters): bank boundary · Bank-specific adapter abstraction with deterministic and fallback implementations. · `src/adapters`
- [Heritage adapter](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/adapters/heritage_bank_adapter.py): deterministic adapter · Bank-specific adapter for Heritage Trust with structured parsing paths. · `src/adapters/heritage_bank_adapter.py`
- [Generic adapter](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/adapters/generic_bank_adapter.py): fallback adapter · Fallback adapter for unknown banks when no bank-specific implementation exists. · `src/adapters/generic_bank_adapter.py`
- [Extractor](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/agent/extractor.py): tiered extraction · Deterministic parse first, then LLM text, then LLM vision fallback. · `src/agent/extractor.py`
- [LLM](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/agent/llm.py): model access · Provider integration for text and vision-assisted extraction fallback. · `src/agent/llm.py`

### Data & persistence

Tenant-scoped relational model, queries, and encrypted credentials.

- [Models](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/db/models.py): relational schema · Tenants, API keys, connections, sync jobs, accounts, transactions, and balances. · `src/db/models.py`
- [Queries](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/db/queries.py): tenant-scoped db access · Application-level query helpers enforcing tenant isolation and idempotent writes. · `src/db/queries.py`
- [Crypto](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/crypto.py): secrets handling · MultiFernet-based encryption and decryption for bank credentials. · `src/core/crypto.py`
- [DB session](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/db/session.py): postgres access · Shared database session and connection management. · `src/db/session.py`

### Platform & deploy

Runtime packaging, orchestration, and infrastructure support.

- [Metrics](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/metrics.py): observability · Structured logging and CloudWatch EMF metrics for runtime visibility. · `src/core/metrics.py`
- [Logging](https://github.com/mayilian/waycore-bank-scraper/blob/main/src/core/logging.py): observability · Structured application logging used across services. · `src/core/logging.py`
- [CDK](https://github.com/mayilian/waycore-bank-scraper/blob/main/deploy/cdk/waycore_stack.py): infrastructure · AWS infrastructure definition split into foundation and app stacks. · `deploy/cdk/waycore_stack.py`

## Connections

- API app → Routes: hosts · The FastAPI app mounts the route set.
- Routes → Auth: authenticates · Routes validate tenant access through API key auth.
- Routes → Operations: delegates · Request handlers defer business logic to shared operations.
- Operations → Queries: reads/writes · Shared logic persists and loads tenant-scoped state via query helpers.
- Queries → Models: maps · Queries operate on the relational schema.
- Queries → DB session: uses · Query helpers use the shared database session.
- Routes → Queries: scoped access · API reads and writes are constrained to tenant-aware helpers.
- CLI → Operations: reuses · CLI and API share the same application logic layer.
- CLI → DB session: direct db · Operational commands can talk directly to Postgres in local use.
- Worker app → Workflow: runs · Worker entrypoint executes the durable workflow.
- Workflow → Steps: orchestrates · Workflow coordinates browser and extraction steps.
- Workflow → Concurrency: limits · Execution is gated by worker and per-bank concurrency controls.
- Workflow → Adapters: adapts · Workflow calls into bank-specific adapter implementations.
- Steps → Stealth: uses · Browser steps rely on stealth helpers.
- Steps → Screenshots: captures · Failures and diagnostics produce screenshots.
- Adapters → Heritage adapter: includes · Heritage Trust is the deterministic example adapter.
- Adapters → Generic adapter: includes · Generic fallback covers unknown banks.
- Heritage adapter → Extractor: parses via · Heritage adapter uses structured extraction logic.
- Extractor → LLM: falls back to · LLM is used only when deterministic parsing fails.
- Workflow → Queries: persists state · Workflow updates sync results, accounts, transactions, and balances.
- Crypto → Workflow: decrypts creds · Worker decrypts bank credentials in memory during execution.
- Auth → Models: validates keys · API key records are stored in the schema.
- Logging → API app: instrumentation · API emits structured logs.
- Metrics → Worker app: instrumentation · Worker emits operational metrics.
- CDK → API app: deploys · Infrastructure provisions the API service.
- CDK → Worker app: deploys · Infrastructure provisions the worker service.
- CDK → DB session: provisions · Infrastructure includes shared Postgres connectivity.
- CDK → Workflow: integrates · Deployment includes Restate-backed workflow routing.
