How mayilian/waycore-bank-scraper works
1. Overall shape
- WayCore is a two-service system built around a shared Postgres database: a FastAPI API for tenant-facing actions and a worker for browser automation.
- The services communicate through Restate, which provides durable workflows, checkpointing, pause/resume, and replay after crashes.
2. Request and control plane
- The API service exposes connection/sync/job/account/transaction endpoints under src/api/routes/.
- It is intentionally thin: routes validate input, authenticate via API key, then delegate to shared business logic in src/core/operations.py and tenant-scoped DB helpers in src/db/queries.py.
- This is the boundary for clients and integrations; the API never runs Playwright itself.
3. Durable sync workflow
- The sync pipeline lives in src/worker/workflow.py and is the core orchestration boundary.
- The workflow is structured as login → extract_all → finalise, with Restate journaling each step so a crash resumes from the last checkpoint rather than starting over.
- OTP handling is designed as a suspend/resume point: the workflow can wait without holding browser resources.
4. Browser automation plane
- The worker service runs Playwright-based browser sessions, stealth helpers, screenshots, and URL normalization logic from src/core/stealth.py, src/core/screenshots.py, and src/core/urls.py.
- The login browser captures authenticated state; the extraction browser reuses that session to visit accounts and collect balances and transactions.
- The worker is the only place where bank portals are touched directly.
5. Bank adapter boundary
- Bank-specific behavior is isolated under src/adapters/.
- BankAdapter is the abstraction, with a deterministic Heritage Trust adapter and a generic fallback for unknown banks.
- This is the main extensibility seam: adding a bank means adding an adapter, not changing workflow or API layers.
6. Extraction strategy
- Extraction is tiered: deterministic DOM parsing first, then LLM text fallback, then LLM vision fallback if needed.
- The LLM machinery in src/agent/llm.py and src/agent/extractor.py is only invoked when selectors or parsing fail, which keeps the happy path fast and cheap.
- The Heritage demo adapter mostly uses pure parsing functions in src/adapters/heritage_parsers.py, so browser automation and data extraction remain separated.
7. Persistence model
- src/db/models.py defines the relational model around tenants, API keys, bank connections, sync jobs, account sync results, accounts, transactions, and balance history.
- Writes are designed to be idempotent: transactions use conflict-avoiding inserts, balances are append-only, and sync results track partial success.
- Migrations under alembic/ represent the schema evolution for sync tracking, URL normalization, deduplication, and API keys.
8. Tenant isolation and auth
- Tenant access is enforced in src/api/auth.py using API keys hashed with SHA-256 and checked with timing-safe comparison.
- All reads and writes are scoped through tenant-aware query helpers in src/db/queries.py; the API layer does not expose raw cross-tenant model access.
- This is a deliberate application-level boundary for multi-tenant safety.
9. Credential and secrets handling
- Bank credentials are encrypted at rest with MultiFernet in src/core/crypto.py.
- Decryption happens only inside worker memory, and key rotation is supported without re-encrypting the database.
- External secrets also include LLM provider keys or AWS IAM-based Bedrock access, depending on deployment configuration.
10. Concurrency and rate protection
- src/worker/concurrency.py enforces both global worker limits and per-bank semaphore limits.
- This protects the worker from browser overload and reduces the risk of bank-side throttling or IP blocking.
- The architecture assumes workflow queueing is preferable to uncontrolled parallel browser launches.
11. CLI and shared application logic
- cli.py provides operational commands for sync execution, OTP submission, and data inspection.
- The CLI and API both reuse the same business logic layer in src/core/operations.py, so the control plane is shared even though the entrypoints differ.
- In local development, the CLI can talk directly to Postgres, while the API path goes through auth and tenant scoping.
12. Runtime and deployment
- The project is packaged as a single Docker image but deployed as two ECS services with different resource profiles: API is lightweight, worker is memory-heavy.
- AWS CDK under deploy/cdk/ splits infrastructure into Foundation and App stacks, with ECR, VPC, RDS Postgres, S3 screenshots, Secrets Manager, and Restate as supporting infrastructure.
- Restate and Cloud Map provide private service-to-service routing inside the VPC.
13. Observability and operational concerns
- src/core/logging.py and src/core/metrics.py provide structured logs and CloudWatch EMF metrics.
- Health checks are exposed through the API, while screenshots and workflow traces help diagnose extraction failures.
- The system is built to be crash-tolerant rather than perfectly synchronous, so observability is tied to workflow steps and DB state.
14. Testing and validation
- Unit tests cover adapters, crypto, extraction, models, and URL handling; integration tests cover API behavior and failure modes against Postgres.
- This mirrors the architecture: pure logic is testable without browsers, while end-to-end behavior depends on workflow, DB, and browser execution together.
Main entry points: workflow.py Workflow · queries.py Queries · waycore_stack.py CDK · routes Routes
Components 23
API & control
Tenant-facing request handling and shared business operations.
- API app · Main HTTP entrypoint for tenant-facing actions and health checks.
src/api/app.py - Routes · Connection, sync, account, transaction, and health endpoints.
src/api/routes - Auth · Tenant authentication and access control based on hashed API keys.
src/api/auth.py - Operations · Core application logic reused by API and CLI entrypoints.
src/core/operations.py - CLI · Operational commands for sync execution, OTP submission, and inspection.
cli.py
Worker & browser
Durable sync workflow and Playwright-based portal automation.
- Workflow · Restate-backed login → extract → finalise pipeline with checkpointing and resume.
src/worker/workflow.py - Steps · Browser and sync step implementations used by the durable workflow.
src/worker/steps.py - Concurrency · Global and per-bank semaphore controls to prevent overload and throttling.
src/worker/concurrency.py - Worker app · Worker process entrypoint that runs automation and workflow execution.
src/worker/app.py - Stealth · Playwright stealth and browser-behavior helpers for portal access.
src/core/stealth.py - Screenshots · Screenshot capture used for debugging and workflow failure analysis.
src/core/screenshots.py - Adapters · Bank-specific adapter abstraction with deterministic and fallback implementations.
src/adapters - Heritage adapter · Bank-specific adapter for Heritage Trust with structured parsing paths.
src/adapters/heritage_bank_adapter.py - Generic adapter · Fallback adapter for unknown banks when no bank-specific implementation exists.
src/adapters/generic_bank_adapter.py - Extractor · Deterministic parse first, then LLM text, then LLM vision fallback.
src/agent/extractor.py - LLM · Provider integration for text and vision-assisted extraction fallback.
src/agent/llm.py
Data & persistence
Tenant-scoped relational model, queries, and encrypted credentials.
- Models · Tenants, API keys, connections, sync jobs, accounts, transactions, and balances.
src/db/models.py - Queries · Application-level query helpers enforcing tenant isolation and idempotent writes.
src/db/queries.py - Crypto · MultiFernet-based encryption and decryption for bank credentials.
src/core/crypto.py - DB session · Shared database session and connection management.
src/db/session.py
Platform & deploy
Runtime packaging, orchestration, and infrastructure support.
Connections 28
- API app → Routes: hosts · The FastAPI app mounts the route set.
- Routes → Auth: authenticates · Routes validate tenant access through API key auth.
- Routes → Operations: delegates · Request handlers defer business logic to shared operations.
- Operations → Queries: reads/writes · Shared logic persists and loads tenant-scoped state via query helpers.
- Queries → Models: maps · Queries operate on the relational schema.
- Queries → DB session: uses · Query helpers use the shared database session.
- Routes → Queries: scoped access · API reads and writes are constrained to tenant-aware helpers.
- CLI → Operations: reuses · CLI and API share the same application logic layer.
- CLI → DB session: direct db · Operational commands can talk directly to Postgres in local use.
- Worker app → Workflow: runs · Worker entrypoint executes the durable workflow.
- Workflow → Steps: orchestrates · Workflow coordinates browser and extraction steps.
- Workflow → Concurrency: limits · Execution is gated by worker and per-bank concurrency controls.
- Workflow → Adapters: adapts · Workflow calls into bank-specific adapter implementations.
- Steps → Stealth: uses · Browser steps rely on stealth helpers.
- Steps → Screenshots: captures · Failures and diagnostics produce screenshots.
- Adapters → Heritage adapter: includes · Heritage Trust is the deterministic example adapter.
- Adapters → Generic adapter: includes · Generic fallback covers unknown banks.
- Heritage adapter → Extractor: parses via · Heritage adapter uses structured extraction logic.
- Extractor → LLM: falls back to · LLM is used only when deterministic parsing fails.
- Workflow → Queries: persists state · Workflow updates sync results, accounts, transactions, and balances.
- Crypto → Workflow: decrypts creds · Worker decrypts bank credentials in memory during execution.
- Auth → Models: validates keys · API key records are stored in the schema.
- Logging → API app: instrumentation · API emits structured logs.
- Metrics → Worker app: instrumentation · Worker emits operational metrics.
- CDK → API app: deploys · Infrastructure provisions the API service.
- CDK → Worker app: deploys · Infrastructure provisions the worker service.
- CDK → DB session: provisions · Infrastructure includes shared Postgres connectivity.
- CDK → Workflow: integrates · Deployment includes Restate-backed workflow routing.