How nuix/kafka-connect-spooldir works
Project type and purpose
• This is an open-source Kafka Connect “Source” plugin (java library) named kafka-connect-spooldir.
• Purpose: watch one or more directories on the file system, parse new files (JSON, CSV, Avro, ELF, binary, line-delimited, schema-less JSON), convert them to Kafka Connect Records, and publish to Kafka topics.
File structure
Top-level:
• bin/ (helper scripts, e.g. debug.sh)
• config/ (example .properties/.json files per connector type)
• docker-compose.yml (local integration testing with Connect & Kafka)
• pom.xml (Maven build + packaging into plugin bundle)
• Jenkinsfile (CI pipeline)
• src/main/java/com/github/jcustenborder/kafka/connect/spooldir/…
– Abstract classes (AbstractSourceConnector, AbstractSourceTask, cleanup policies, schema generator)
– Concrete connectors and tasks per data format (SpoolDirJsonSourceConnector, SpoolDirCsvSourceConnector, SpoolDirAvroSourceConnector, SpoolDirELFSourceConnector, SpoolDirBinaryFileSourceConnector, SpoolDirLineDelimitedSourceConnector, SpoolDirSchemaLessJsonSourceConnector)
– Format-specific schema generators, file comparators, partitioner predicates
• src/test/java/… + src/test/resources/… (unit and integration tests for each connector, for various formats and cleanup policies)
README for additional insights
• Detailed configuration options for each connector type
• Links to online documentation and Confluent Hub packaging
• Maven build, how to assemble the plugin under target/kafka-connect-target for deployment into Connect worker plugin.path
Detailed capture of essential elements
• Concrete connector classes under Connector Coordinator (one per format)
• How AbstractSchemaGenerator can be invoked standalone (e.g., via mvn exec) for generating .avsc schemas from sample data
• Deployment: plugin.zip extracted under plugin.path on Connect worker
• Local testing via docker-compose (Connect REST API, Kafka broker)