# nuix/kafka-connect-spooldir architecture

> Architecture diagram of the GitHub repository nuix/kafka-connect-spooldir, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nuix/kafka-connect-spooldir
- Repository: https://github.com/nuix/kafka-connect-spooldir

## Overview

##### Project type and purpose
- This is an open-source Kafka Connect “Source” plugin (java library) named kafka-connect-spooldir.
- Purpose: watch one or more directories on the file system, parse new files (JSON, CSV, Avro, ELF, binary, line-delimited, schema-less JSON), convert them to Kafka Connect Records, and publish to Kafka topics.

##### File structure
Top-level:
- bin/ (helper scripts, e.g. debug.sh)
- config/ (example .properties/.json files per connector type)
- docker-compose.yml (local integration testing with Connect & Kafka)
- pom.xml (Maven build + packaging into plugin bundle)
- Jenkinsfile (CI pipeline)
- src/main/java/com/github/jcustenborder/kafka/connect/spooldir/…
– Abstract classes (AbstractSourceConnector, AbstractSourceTask, cleanup policies, schema generator)
– Concrete connectors and tasks per data format (SpoolDirJsonSourceConnector, SpoolDirCsvSourceConnector, SpoolDirAvroSourceConnector, SpoolDirELFSourceConnector, SpoolDirBinaryFileSourceConnector, SpoolDirLineDelimitedSourceConnector, SpoolDirSchemaLessJsonSourceConnector)
– Format-specific schema generators, file comparators, partitioner predicates
- src/test/java/… + src/test/resources/… (unit and integration tests for each connector, for various formats and cleanup policies)

##### README for additional insights
- Detailed configuration options for each connector type
- Links to online documentation and Confluent Hub packaging
- Maven build, how to assemble the plugin under target/kafka-connect-target for deployment into Connect worker plugin.path

##### Detailed capture of essential elements
- Concrete connector classes under Connector Coordinator (one per format)
- How AbstractSchemaGenerator can be invoked standalone (e.g., via mvn exec) for generating .avsc schemas from sample data
- Deployment: plugin.zip extracted under plugin.path on Connect worker
- Local testing via docker-compose (Connect REST API, Kafka broker)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "File System"
        direction TB
        FS_Input["Input Directory"]:::filesystem
        FS_Error["Error Directory"]:::filesystem
        FS_Finished["Finished Directory"]:::filesystem
    end

    subgraph "Kafka Connect Worker" 
        direction TB
        Loader["Plugin Loader"]:::connect
        Coordinator["Connector Coordinator"]:::connect
        Executor["Task Executor"]:::connect
        subgraph "Strategy Modules"
            direction TB
            SchemaGen["Schema Generator"]:::connect
            CleanupPolicy["Cleanup Policy"]:::connect
            Partitioner["Task Partitioner"]:::connect
        end
        ConnectorClasses["Concrete Connectors & Tasks"]:::connect
    end

    subgraph "Kafka Cluster"
        direction TB
        Brokers["Brokers"]:::kafka
        Topics["Topics"]:::kafka
    end

    subgraph "CI/CD & Packaging"
        direction TB
        Maven["Maven / Confluent Hub"]:::cicd
        DockerCompose["Local Integration Testing"]:::cicd
        Scripts["Helper Scripts"]:::cicd
        Configs["Example Connector Configs"]:::cicd
    end

    FS_Input -->|"watch & read file"| Executor
    Executor -->|"emit ConnectRecord"| Loader
    Loader --> Coordinator
    Coordinator --> Executor
    Executor -->|apply cleanup policy| FS_Finished
    Executor -->|on error| FS_Error
    Executor -->|send to KafkaProducer| Brokers
    Brokers --> Topics

    Maven -->|build & package| DockerCompose
    DockerCompose -->|tests with| Brokers
    Scripts -.-> Maven
    Configs -.-> Executor

    click pom.xml "https://github.com/nuix/kafka-connect-spooldir/blob/master/pom.xml"
    click docker-compose.yml "https://github.com/nuix/kafka-connect-spooldir/blob/master/docker-compose.yml"
    click Scripts "https://github.com/nuix/kafka-connect-spooldir/blob/master/bin/debug.sh"
    click Configs "https://github.com/nuix/kafka-connect-spooldir/blob/master/config/*.properties"
    click FS_Input "https://github.com/nuix/kafka-connect-spooldir/blob/master/config/*.properties"
    click FS_Input "https://github.com/nuix/kafka-connect-spooldir/blob/master/config/*.json"
    click Loader "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractSourceConnector.java"
    click Loader "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractSpoolDirSourceConnector.java"
    click Executor "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractSourceTask.java"
    click Executor "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractSpoolDirSourceTask.java"
    click Configs "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractSourceConnectorConfig.java"
    click Configs "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractSpoolDirSourceConnectorConfig.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirCsvSourceConnector.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirCsvSourceConnectorConfig.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirCsvSourceTask.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirJsonSourceConnector.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirJsonSourceConnectorConfig.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirJsonSourceTask.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirAvroSourceConnector.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirAvroSourceConnectorConfig.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirAvroSourceTask.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/elf/SpoolDirELFSourceConnector.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/elf/SpoolDirELFSourceConnectorConfig.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/elf/SpoolDirELFSourceTask.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirBinaryFileSourceConnector.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirBinaryFileSourceConnectorConfig.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirBinaryFileSourceTask.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirLineDelimitedSourceConnector.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirLineDelimitedSourceConnectorConfig.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirLineDelimitedSourceTask.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirSchemaLessJsonSourceConnector.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirSchemaLessJsonSourceConnectorConfig.java"
    click ConnectorClasses "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/SpoolDirSchemaLessJsonSourceTask.java"
    click SchemaGen "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/CsvSchemaGenerator.java"
    click SchemaGen "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/JsonSchemaGenerator.java"
    click SchemaGen "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractSchemaGenerator.java"
    click SchemaGen "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/elf/SchemaConversionBuilder.java"
    click CleanupPolicy "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractCleanUpPolicy.java"
    click Partitioner "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/FileComparator.java"
    click Partitioner "https://github.com/nuix/kafka-connect-spooldir/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/spooldir/AbstractTaskPartitionerPredicate.java"
    classDef filesystem fill:#FFF59D,stroke:#FBC02D,color:#000
    classDef connect fill:#BBDEFB,stroke:#1976D2,color:#000
    classDef kafka fill:#C8E6C9,stroke:#388E3C,color:#000
    classDef cicd fill:#E0E0E0,stroke:#616161,color:#000
```
