How daftyon/easier-batch works
Project Type & Purpose
• It provides a Reader-Processor-Writer pipeline, transaction management, retries/skips, metadata persistence (MySQL), monitoring, and pluggable extensions.
File Structure Highlights
• core/
• item_reader.py, item_processor.py, item_writer.py – abstract base classes and core interfaces
• step.py, job.py, job_launcher.py – orchestration of batch steps and jobs
• metadata_manager.py – manages the JDBC/ORM connection to the metadata tables
• setup.py / easier_batch.egg-info / dist/ – packaging and distribution artifacts
• test_*.py – automated tests for core behavior and database integration
• batch.py – a sample or entry-point script
Key Architectural Patterns
• Layered Architecture (Presentation / Business / Infrastructure / Core Engine / Components / Data)
• Reader-Processor-Writer chain inside each Step
• Job → multiple Steps → each Step runs Read → Process → Write with transaction boundaries
• Pluggable implementations (FileReader, DatabaseReader, APIReader, KafkaReader, etc.)
• Cross-cutting concerns (retry/skip, transaction management, monitoring/logging)
Main Components & Relationships
A. Presentation Layer
• CLI (batch.py)
• (Optional) REST API or Web Dashboard extensions
↓ invokes
B. Business Layer
• JobLauncher – loads configuration, validates, and kicks off a Job
↓ orchestrates
C. Core Engine
1) Job Execution Engine
– Manages one or more Job instances
– Orchestrates parallel/sequential Step execution
2) Step Execution Engine
– For each Step: opens transaction, calls Chunk Manager, commits/rolls back
↓ within each Step
D. Component Layer
• ItemReader – reads data in chunks
• ItemProcessor – transforms, filters, validates items
• ItemWriter – persists output items
These three form a pipeline:
ItemReader → ItemProcessor → ItemWriter
↑ error handling / retry / skip plugged in at each boundary
E. Infrastructure Layer
• MetadataManager – persists job/step metadata in MySQL tables
• Transaction Manager – JDBC connection pooling, commit/rollback logic
• Retry & Skip Policies – configured via BatchConfiguration
• Monitoring & Logging – collects metrics, health checks, and audit trails
F. Data Layer
• MySQL metadata tables (job_metadata, step_metadata, execution_context)
• (Optional) external data sources: file system, Kafka, REST APIs