How flow-php/etl works
Project Type and Purpose
• This is an open-source PHP library/framework for ETL (Extract, Transform, Load) data pipelines.
• It provides a core Flow API plus modular adapters (extractors, transformers, loaders), configuration builders, caching, error handling, retry strategies, and schema/validation support.
• Its goal is to let developers build memory-efficient, strongly typed, generator-based data flows across diverse sources and sinks.
High-Level Architecture Components
a. Core Flow Engine
• Flow\ETL\FlowContext & Flow\ETL\Flow – orchestration of the pipeline stages
• Flow\ETL\Pipeline / PipelineFactory – constructs the sequence of pipeline steps
• Flow\ETL\Extractor / Transformer / Loader – abstract base interfaces and implementations
• Flow\ETL\DataFrame / Row / Schema – in-memory representation of tabular data and its metadata
b. Extraction Layer
• Extractors (ArrayExtractor, FileExtractor, CacheExtractor, GeneratorExtractor, etc.)
• SequenceGenerators (NumberSequenceGenerator, DatePeriodSequenceGenerator)
• PathPartitionsExtractor, ChunkExtractor
c. Transformation Layer
• Transformers (AutoCastTransformer, RenameEntryTransformer, WindowFunctionTransformer, etc.)
• DSL Functions (Flow\ETL\Function\*) providing single-row operations
• Constraint and Config modules for sort, cache, and unique constraints
d. Loading Layer
• Loaders (ArrayLoader, FileLoader, StreamLoader, CallbackLoader, SchemaValidationLoader, etc.)
• Serializer abstractions (Base64Serializer, NativePHPSerializer, CompressingSerializer)
e. Support Services
• Cache implementations (FilesystemCache, InMemoryCache, PSRSimpleCache)
• Retry and Delay strategies (Fixed, Exponential, Jitter, Linear)
• Clock & Sleep interfaces for testability (SystemClock, FakeClock, SystemSleep, FakeSleep)
• Error Handlers (IgnoreError, SkipRows, ThrowError)
• Schema Formatters & Validators (JSON, PHP, ASCII)
Component Interactions and Data Flow
• A Flow instance is configured with one or more Extractors → Pipelines → Transformers → Loaders.
• Extractors produce Row streams or DataFrames (via generators).
• Pipelines chain together transform and sort operations (BatchingPipeline, HashJoinPipeline, PartitioningPipeline, etc.).
• Transformers mutate or enrich each Row/DataFrame chunk.
• Loaders consume the final Row/DataFrame to write to destination (memory, file, database).
• Caching, retrying, and error handling wrap around extract/transform/load steps as decorators/branches.
Architectural Patterns and Design Principles
• Strategy Pattern: swap Extractor, Transformer, Loader, RetryStrategy, DelayFactory implementations
• Builder Pattern: ConfigBuilder, CacheConfigBuilder, SortConfigBuilder assemble pipeline options
• Chain of Responsibility / Decorator: pipelines, extractors, loaders wrap around each other
• Factory Pattern: PipelineFactory, DataFrameFactory, EntryFactory produce typed objects
• Single Responsibility: clear separation between extraction, transformation, loading, and configuration
• Dependency Inversion: interfaces for clock, filesystem, cache, retry services allow injection/mockability
Essential Details to Capture
• Flow entry point and FlowContext driving the pipeline
• DataFrame/Row lifecycle as it flows through stages
• Configuration builders influencing cache/sort behavior
• That extractors and loaders can be chained or branched
• Retry and error handling wrappers on extract/load steps