# vgpastor/batchactions architecture

> Architecture diagram of the GitHub repository vgpastor/batchactions, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vgpastor/batchactions
- Repository: https://github.com/vgpastor/batchactions

## Overview

@batchactions is a set of TypeScript libraries for record-by-record batch processing, with optional import validation, distributed workers, and pluggable persistence. The principal workflow configures a job, reads records from a source, processes them in batches, and tracks job state and lifecycle events. Import workflows add parsing, schema validation, and preview; distributed workflows add batch claiming and worker processing. README capabilities are represented where source excerpts are incomplete; uncertain internal wiring is omitted.

Main entry points: [`packages/core/src/BatchEngine.ts`](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/BatchEngine.ts) (BatchEngine), [`packages/import/src/BulkImport.ts`](https://github.com/vgpastor/batchactions/blob/main/packages/import/src/BulkImport.ts) (BulkImport), [`packages/import/src/application/usecases/PreviewImport.ts`](https://github.com/vgpastor/batchactions/blob/main/packages/import/src/application/usecases/PreviewImport.ts) (Import preview), [`packages/core/src/domain/ports/StateStore.ts`](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/domain/ports/StateStore.ts) (State store port)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core processing"]
  node_engine["BatchEngine<br/>[BatchEngine.ts]"]
  node_source["Data sources"]
  node_usecases["Job use cases"]
  node_context["Job context<br/>[JobContext.ts]"]
  node_events["Event bus<br/>[EventBus.ts]"]
  node_stateport["State store port<br/>[StateStore.ts]"]
  node_distport["Distributed store port"]
end

subgraph group_import["Import workflow"]
  node_importer["BulkImport<br/>[BulkImport.ts]"]
  node_parsers["Format parsers"]
  node_preview["Import preview<br/>[PreviewImport.ts]"]
  node_validator["Schema validator<br/>[SchemaValidator.ts]"]
end

subgraph group_distributed["Distributed workers"]
  node_distributed["Distributed import"]
  node_prepare["Prepare batches"]
  node_worker["Process batch"]
end

subgraph group_persistence["State persistence"]
  node_memory[("Memory store")]
  node_filestore[("File store<br/>[FileStateStore.ts]")]
  node_sequelize[("Sequelize adapter")]
  node_prisma[("Prisma adapter")]
end

node_user(("Application"))
node_processor(("Record processor"))
node_database[("SQL database")]

node_user -->|"configures"| node_engine
node_engine -->|"reads records"| node_source
node_engine -->|"delegates"| node_usecases
node_engine -->|"creates context"| node_context
node_usecases -->|"processes records"| node_processor
node_context -->|"saves state"| node_stateport
node_context -->|"emits events"| node_events
node_user -->|"configures import"| node_importer
node_importer -->|"reads records"| node_source
node_importer -->|"parses data"| node_parsers
node_importer -->|"validates records"| node_validator
node_user -->|"requests preview"| node_preview
node_preview -->|"reads chunks"| node_source
node_preview -->|"parses chunks"| node_parsers
node_preview -->|"validates sample"| node_validator
node_user -->|"starts distributed job"| node_distributed
node_distributed -->|"prepares batches"| node_prepare
node_distributed -->|"dispatches batches"| node_worker
node_worker -.->|"validates records"| node_validator
node_worker -->|"claims and saves"| node_distport
node_engine -->|"uses store"| node_stateport
node_memory -->|"implements"| node_stateport
node_filestore -->|"implements"| node_stateport
node_sequelize -->|"implements"| node_distport
node_prisma -->|"implements"| node_distport
node_sequelize -->|"reads and writes"| node_database
node_prisma -->|"reads and writes"| node_database

click node_engine "https://github.com/vgpastor/batchactions/blob/main/packages/core/src/BatchEngine.ts"
click node_source "https://github.com/vgpastor/batchactions/tree/main/packages/core/src/infrastructure/sources"
click node_usecases "https://github.com/vgpastor/batchactions/tree/main/packages/core/src/application/usecases"
click node_context "https://github.com/vgpastor/batchactions/blob/main/packages/core/src/application/JobContext.ts"
click node_events "https://github.com/vgpastor/batchactions/blob/main/packages/core/src/application/EventBus.ts"
click node_importer "https://github.com/vgpastor/batchactions/blob/main/packages/import/src/BulkImport.ts"
click node_parsers "https://github.com/vgpastor/batchactions/tree/main/packages/import/src/infrastructure/parsers"
click node_preview "https://github.com/vgpastor/batchactions/blob/main/packages/import/src/application/usecases/PreviewImport.ts"
click node_validator "https://github.com/vgpastor/batchactions/blob/main/packages/import/src/domain/services/SchemaValidator.ts"
click node_distributed "https://github.com/vgpastor/batchactions/blob/main/packages/distributed/src/DistributedImport.ts"
click node_prepare "https://github.com/vgpastor/batchactions/blob/main/packages/distributed/src/PrepareDistributedImport.ts"
click node_worker "https://github.com/vgpastor/batchactions/blob/main/packages/distributed/src/ProcessDistributedBatch.ts"
click node_stateport "https://github.com/vgpastor/batchactions/blob/main/packages/core/src/domain/ports/StateStore.ts"
click node_distport "https://github.com/vgpastor/batchactions/blob/main/packages/core/src/domain/ports/DistributedStateStore.ts"
click node_memory "https://github.com/vgpastor/batchactions/blob/main/packages/core/src/infrastructure/state/InMemoryStateStore.ts"
click node_filestore "https://github.com/vgpastor/batchactions/blob/main/packages/core/src/infrastructure/state/FileStateStore.ts"
click node_sequelize "https://github.com/vgpastor/batchactions/blob/main/packages/state-sequelize/src/SequelizeStateStore.ts"
click node_prisma "https://github.com/vgpastor/batchactions/blob/main/packages/state-prisma/src/PrismaStateStore.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_source,node_usecases,node_context,node_events,node_stateport,node_distport toneBlue
class node_importer,node_parsers,node_preview,node_validator,node_database toneAmber
class node_distributed,node_prepare,node_worker toneMint
class node_memory,node_filestore,node_sequelize,node_prisma toneRose
class node_user,node_processor toneIndigo
```

## Components

### Core processing

- [BatchEngine](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/BatchEngine.ts): `packages/core/src/BatchEngine.ts`
- [Data sources](https://github.com/vgpastor/batchactions/tree/main/packages/core/src/infrastructure/sources): `packages/core/src/infrastructure/sources`
- [Job use cases](https://github.com/vgpastor/batchactions/tree/main/packages/core/src/application/usecases): `packages/core/src/application/usecases`
- [Job context](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/application/JobContext.ts): `packages/core/src/application/JobContext.ts`
- [Event bus](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/application/EventBus.ts): `packages/core/src/application/EventBus.ts`
- [State store port](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/domain/ports/StateStore.ts): `packages/core/src/domain/ports/StateStore.ts`
- [Distributed store port](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/domain/ports/DistributedStateStore.ts): `packages/core/src/domain/ports/DistributedStateStore.ts`

### Import workflow

- [BulkImport](https://github.com/vgpastor/batchactions/blob/main/packages/import/src/BulkImport.ts): `packages/import/src/BulkImport.ts`
- [Format parsers](https://github.com/vgpastor/batchactions/tree/main/packages/import/src/infrastructure/parsers): `packages/import/src/infrastructure/parsers`
- [Import preview](https://github.com/vgpastor/batchactions/blob/main/packages/import/src/application/usecases/PreviewImport.ts): `packages/import/src/application/usecases/PreviewImport.ts`
- [Schema validator](https://github.com/vgpastor/batchactions/blob/main/packages/import/src/domain/services/SchemaValidator.ts): `packages/import/src/domain/services/SchemaValidator.ts`

### Distributed workers

- [Distributed import](https://github.com/vgpastor/batchactions/blob/main/packages/distributed/src/DistributedImport.ts): `packages/distributed/src/DistributedImport.ts`
- [Prepare batches](https://github.com/vgpastor/batchactions/blob/main/packages/distributed/src/PrepareDistributedImport.ts): `packages/distributed/src/PrepareDistributedImport.ts`
- [Process batch](https://github.com/vgpastor/batchactions/blob/main/packages/distributed/src/ProcessDistributedBatch.ts): `packages/distributed/src/ProcessDistributedBatch.ts`

### State persistence

- [Memory store](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/infrastructure/state/InMemoryStateStore.ts): `packages/core/src/infrastructure/state/InMemoryStateStore.ts`
- [File store](https://github.com/vgpastor/batchactions/blob/main/packages/core/src/infrastructure/state/FileStateStore.ts): `packages/core/src/infrastructure/state/FileStateStore.ts`
- [Sequelize adapter](https://github.com/vgpastor/batchactions/blob/main/packages/state-sequelize/src/SequelizeStateStore.ts): `packages/state-sequelize/src/SequelizeStateStore.ts`
- [Prisma adapter](https://github.com/vgpastor/batchactions/blob/main/packages/state-prisma/src/PrismaStateStore.ts): `packages/state-prisma/src/PrismaStateStore.ts`

### Other components

- **Application**
- **Record processor**
- **SQL database**

## Connections

- Application → BatchEngine: configures
- BatchEngine → Data sources: reads records
- BatchEngine → Job use cases: delegates
- BatchEngine → Job context: creates context
- Job use cases → Record processor: processes records
- Job context → State store port: saves state
- Job context → Event bus: emits events
- Application → BulkImport: configures import
- BulkImport → Data sources: reads records
- BulkImport → Format parsers: parses data
- BulkImport → Schema validator: validates records
- Application → Import preview: requests preview
- Import preview → Data sources: reads chunks
- Import preview → Format parsers: parses chunks
- Import preview → Schema validator: validates sample
- Application → Distributed import: starts distributed job
- Distributed import → Prepare batches: prepares batches
- Distributed import → Process batch: dispatches batches
- Process batch → Schema validator: validates records
- Process batch → Distributed store port: claims and saves
- BatchEngine → State store port: uses store
- Memory store → State store port: implements
- File store → State store port: implements
- Sequelize adapter → Distributed store port: implements
- Prisma adapter → Distributed store port: implements
- Sequelize adapter → SQL database: reads and writes
- Prisma adapter → SQL database: reads and writes