# flow-php/etl architecture

> Architecture diagram of the GitHub repository flow-php/etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/flow-php/etl
- Repository: https://github.com/flow-php/etl

## Overview

##### Project Type and Purpose
- This is an open-source PHP library/framework for ETL (Extract, Transform, Load) data pipelines.
- It provides a core Flow API plus modular adapters (extractors, transformers, loaders), configuration builders, caching, error handling, retry strategies, and schema/validation support.
- Its goal is to let developers build memory-efficient, strongly typed, generator-based data flows across diverse sources and sinks.

##### High-Level Architecture Components
a. Core Flow Engine
- Flow\ETL\FlowContext & Flow\ETL\Flow – orchestration of the pipeline stages
- Flow\ETL\Pipeline / PipelineFactory – constructs the sequence of pipeline steps
- Flow\ETL\Extractor / Transformer / Loader – abstract base interfaces and implementations
- Flow\ETL\DataFrame / Row / Schema – in-memory representation of tabular data and its metadata
b. Extraction Layer
- Extractors (ArrayExtractor, FileExtractor, CacheExtractor, GeneratorExtractor, etc.)
- SequenceGenerators (NumberSequenceGenerator, DatePeriodSequenceGenerator)
- PathPartitionsExtractor, ChunkExtractor
c. Transformation Layer
- Transformers (AutoCastTransformer, RenameEntryTransformer, WindowFunctionTransformer, etc.)
- DSL Functions (Flow\ETL\Function\*) providing single-row operations
- Constraint and Config modules for sort, cache, and unique constraints
d. Loading Layer
- Loaders (ArrayLoader, FileLoader, StreamLoader, CallbackLoader, SchemaValidationLoader, etc.)
- Serializer abstractions (Base64Serializer, NativePHPSerializer, CompressingSerializer)
e. Support Services
- Cache implementations (FilesystemCache, InMemoryCache, PSRSimpleCache)
- Retry and Delay strategies (Fixed, Exponential, Jitter, Linear)
- Clock & Sleep interfaces for testability (SystemClock, FakeClock, SystemSleep, FakeSleep)
- Error Handlers (IgnoreError, SkipRows, ThrowError)
- Schema Formatters & Validators (JSON, PHP, ASCII)

##### Component Interactions and Data Flow
- A Flow instance is configured with one or more Extractors → Pipelines → Transformers → Loaders.
- Extractors produce Row streams or DataFrames (via generators).
- Pipelines chain together transform and sort operations (BatchingPipeline, HashJoinPipeline, PartitioningPipeline, etc.).
- Transformers mutate or enrich each Row/DataFrame chunk.
- Loaders consume the final Row/DataFrame to write to destination (memory, file, database).
- Caching, retrying, and error handling wrap around extract/transform/load steps as decorators/branches.

##### Architectural Patterns and Design Principles
- Strategy Pattern: swap Extractor, Transformer, Loader, RetryStrategy, DelayFactory implementations
- Builder Pattern: ConfigBuilder, CacheConfigBuilder, SortConfigBuilder assemble pipeline options
- Chain of Responsibility / Decorator: pipelines, extractors, loaders wrap around each other
- Factory Pattern: PipelineFactory, DataFrameFactory, EntryFactory produce typed objects
- Single Responsibility: clear separation between extraction, transformation, loading, and configuration
- Dependency Inversion: interfaces for clock, filesystem, cache, retry services allow injection/mockability

##### Essential Details to Capture
- Flow entry point and FlowContext driving the pipeline
- DataFrame/Row lifecycle as it flows through stages
- Configuration builders influencing cache/sort behavior
- That extractors and loaders can be chained or branched
- Retry and error handling wrappers on extract/load steps

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Flow Engine
    subgraph "Core Flow Engine"
        direction TB
        Flow["Flow"]:::core
        FlowContext["FlowContext"]:::core
    end

    %% Pipeline
    PipelineInterface["Pipeline Interface"]:::core
    subgraph "Pipeline Implementations"
        direction TB
        BatchingPipeline["BatchingPipeline"]:::core
        HashJoinPipeline["HashJoinPipeline"]:::core
        PartitioningPipeline["PartitioningPipeline"]:::core
    end

    %% Extractors
    subgraph "Extraction Layer"
        direction TB
        ExtractorInterface["Extractor Interface"]:::extractor
        subgraph "Extractor Implementations"
            direction TB
            ArrayExtractor["ArrayExtractor"]:::extractor
            FileExtractor["FileExtractor"]:::extractor
            CacheExtractor["CacheExtractor"]:::extractor
            GeneratorExtractor["GeneratorExtractor"]:::extractor
            PathPartitionsExtractor["PathPartitionsExtractor"]:::extractor
            ChunkExtractor["ChunkExtractor"]:::extractor
        end
        subgraph "Sequence Generators"
            direction TB
            NumberSequenceGenerator["NumberSequenceGenerator"]:::extractor
            DatePeriodSequenceGenerator["DatePeriodSequenceGenerator"]:::extractor
        end
    end

    %% Data Structures
    DataFrame["DataFrame / Row"]:::core
    Schema["Schema & Validation"]:::core

    %% Transformers
    subgraph "Transformation Layer"
        direction TB
        TransformerInterface["Transformer Interface"]:::transformer
        DSLFunctions["DSL Functions"]:::transformer
        subgraph "Transformer Implementations"
            direction TB
            AutoCastTransformer["AutoCastTransformer"]:::transformer
            RenameEntryTransformer["RenameEntryTransformer"]:::transformer
            WindowFunctionTransformer["WindowFunctionTransformer"]:::transformer
        end
    end

    %% Loaders
    subgraph "Loading Layer"
        direction TB
        LoaderInterface["Loader Interface"]:::loader
        subgraph "Loader Implementations"
            direction TB
            ArrayLoader["ArrayLoader"]:::loader
            FileLoader["FileLoader"]:::loader
            StreamLoader["StreamLoader"]:::loader
            CallbackLoader["CallbackLoader"]:::loader
            SchemaValidationLoader["SchemaValidationLoader"]:::loader
        end
    end

    %% Support Services
    subgraph "Support Services"
        direction TB
        Cache["Cache"]:::support
        Retry["Retry Strategies"]:::support
        ErrorHandler["Error Handlers"]:::support
        ClockSleep["Clock & Sleep"]:::support
        Serializer["Serializers"]:::support
    end

    %% Interactions
    Flow --> FlowContext
    Flow --> ExtractorInterface
    Flow --> PipelineInterface
    Flow --> TransformerInterface
    Flow --> LoaderInterface

    ExtractorInterface --> ArrayExtractor
    ExtractorInterface --> FileExtractor
    ExtractorInterface --> CacheExtractor
    ExtractorInterface --> GeneratorExtractor
    ExtractorInterface --> PathPartitionsExtractor
    ExtractorInterface --> ChunkExtractor
    ExtractorInterface --> NumberSequenceGenerator
    ExtractorInterface --> DatePeriodSequenceGenerator

    PipelineInterface --> BatchingPipeline
    PipelineInterface --> HashJoinPipeline
    PipelineInterface --> PartitioningPipeline

    TransformerInterface --> AutoCastTransformer
    TransformerInterface --> RenameEntryTransformer
    TransformerInterface --> WindowFunctionTransformer
    TransformerInterface --> DSLFunctions

    LoaderInterface --> ArrayLoader
    LoaderInterface --> FileLoader
    LoaderInterface --> StreamLoader
    LoaderInterface --> CallbackLoader
    LoaderInterface --> SchemaValidationLoader

    ArrayExtractor --> DataFrame
    FileExtractor --> DataFrame
    DataFrame --> PipelineInterface
    BatchingPipeline --> TransformerInterface
    DSLFunctions --> TransformerInterface
    TransformerInterface --> LoaderInterface
    LoaderInterface --> Serializer

    %% Decorators
    Cache --> ExtractorInterface
    Retry --> ExtractorInterface
    ErrorHandler --> ExtractorInterface
    Cache --> LoaderInterface
    Retry --> LoaderInterface
    ErrorHandler --> LoaderInterface
    ClockSleep --> Retry
    ClockSleep --> Cache
    Serializer --> LoaderInterface
    DataFrame --> Schema

    %% Click Events
    click Flow "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Flow.php"
    click FlowContext "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/FlowContext.php"
    click PipelineInterface "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Pipeline.php"
    click BatchingPipeline "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Pipeline/BatchingPipeline.php"
    click HashJoinPipeline "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Pipeline/HashJoinPipeline.php"
    click PartitioningPipeline "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Pipeline/PartitioningPipeline.php"
    click ExtractorInterface "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor.php"
    click ArrayExtractor "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor/ArrayExtractor.php"
    click FileExtractor "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor/FileExtractor.php"
    click CacheExtractor "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor/CacheExtractor.php"
    click GeneratorExtractor "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor/GeneratorExtractor.php"
    click PathPartitionsExtractor "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor/PathPartitionsExtractor.php"
    click ChunkExtractor "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor/ChunkExtractor.php"
    click NumberSequenceGenerator "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor/SequenceGenerator/NumberSequenceGenerator.php"
    click DatePeriodSequenceGenerator "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Extractor/SequenceGenerator/DatePeriodSequenceGenerator.php"
    click DataFrame "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/DataFrame.php"
    click Schema "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Schema.php"
    click TransformerInterface "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Transformer.php"
    click AutoCastTransformer "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Transformer/AutoCastTransformer.php"
    click RenameEntryTransformer "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Transformer/RenameEntryTransformer.php"
    click WindowFunctionTransformer "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Transformer/WindowFunctionTransformer.php"
    click DSLFunctions "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Function/functions.php"
    click LoaderInterface "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Loader.php"
    click ArrayLoader "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Loader/ArrayLoader.php"
    click FileLoader "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Loader/FileLoader.php"
    click StreamLoader "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Loader/StreamLoader.php"
    click CallbackLoader "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Loader/CallbackLoader.php"
    click SchemaValidationLoader "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Loader/SchemaValidationLoader.php"
    click Serializer "https://github.com/flow-php/etl/blob/1.x/src/Flow/Serializer/Serializer.php"
    click Cache "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Cache.php"
    click Retry "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Retry/RetryStrategy.php"
    click ErrorHandler "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/ErrorHandler.php"
    click ClockSleep "https://github.com/flow-php/etl/blob/1.x/src/Flow/Clock/SystemClock.php"
    click ClockSleep "https://github.com/flow-php/etl/blob/1.x/src/Flow/ETL/Time/SystemSleep.php"

    %% Styles
    classDef core fill:#D0E6F8,stroke:#0366d6,stroke-width:1px
    classDef extractor fill:#DFF6DD,stroke:#2C9E27,stroke-width:1px
    classDef transformer fill:#FFE8B3,stroke:#d68d00,stroke-width:1px
    classDef loader fill:#E5D0F2,stroke:#6f42c1,stroke-width:1px
    classDef support fill:#E7E7E7,stroke:#888888,stroke-width:1px
```
