How absaoss/abris works
Project type and purpose
• This is a Scala-based library (not an end-user application) that provides seamless integration between Apache Spark and Apache Avro, including native Avro, Confluent Avro, and Schema Registry support. It exposes two main Spark SQL expressions (to_avro, from_avro), a fluent configuration API (AbrisConfig), SPI extensibility points (custom SchemaConverter implementations), error-handling strategies, and utilities for schema conversion and management.
File structure
• Top-level build artifacts: pom.xml, scalastyle-config.xml, GitHub workflows for CI, code coverage, compatibility checks.
• src/main/scala/org/apache/spark/sql/avro: Core Spark UDF definitions (AbrisAvroSerializer, AbrisAvroDeserializer).
• src/main/scala/za/co/absa/abris/avro: Modular sub-packages:
• errors – exception handler implementations (FailFast, Permissive, SpecificRecord).
• format – conversions between Spark internal types and Avro bytes.
• parsing/utils – Avro schema utility functions.
• read/confluent – Confluent format constants and schema manager factory.
• registry – Schema Registry client abstractions and implementations (mock, Confluent).
• sql – SQL-level hooks for converting Avro ↔ Spark Row and default/custom schema converters.
• src/main/avro – example Avro schemas for testing/documentation.
• src/main/resources/META-INF/services – SPI registration for custom SchemaConverter.
• src/main/scala/za/co/absa/abris/config – centralized configuration mapping for to/from Avro use cases.
• src/main/scala/za/co/absa/abris/examples – executable examples (Kafka reader/writer, data generators).
• src/test – comprehensive unit tests mirroring all modules and entry points.
README for additional insights
• Confirms the library’s two main entry points (to_avro, from_avro), four config flows (toSimpleAvro, toConfluentAvro, fromSimpleAvro, fromConfluentAvro), the use of Confluent Schema Registry, authentication flexibility, and error handling strategies.
• Points to extended documentation in documentation/*.md.
a. Main components
• Data sources/sinks: Kafka topics, Parquet/HDFS files, raw Avro streams.
• Spark SQL engine & runtime (cluster).
• ABRiS library layers:
– SQL expressions (to_avro, from_avro)
– Fluent AbrisConfig builder (to/from, naming strategies, registry URLs, auth configs, exception handlers, converter selection)
– Core modules: Serializer, Deserializer, AvroSchemaUtils, SparkAvroConversions
– Registry client abstraction & implementations
– SchemaManager and SchemaManagerFactory
– SPI loader for custom SchemaConverter
– Exception handler strategies
• Confluent Schema Registry (REST endpoint)
• Optional custom extensions (user-provided SchemaConverter, custom ExceptionHandler)
• Build/packaging: Maven coordinates and dependencies (Spark, spark-avro, abris)
b. Relationships and interactions
1. User Spark job pulls binary Avro data (Kafka/HDFS) into DataFrame.
2. Job invokes from_avro(column, AbrisConfig…) → SQL expression layer.
3. Expression delegates to Deserializer module → requests Avro schema:
• If Confluent mode → uses SchemaManager → uses RegistryClient to call Schema Registry REST API.
• If simple mode → uses provided static schema or auto-generated via AvroSchemaUtils.
4. SchemaConverter (default or SPI custom) maps Avro Schema → Spark StructType.
5. Deserializer produces Spark Rows.
6. For to_avro, the reverse: Spark Row → Catalyst internal → Avro record → binary data, optional prepended Confluent schema id.
7. Error handlers intercept serialization/deserialization exceptions and apply configured strategy.
c. Architectural patterns / design principles
• Layered modular architecture separating concerns (SQL API, core conversion logic, registry access, error handling).
• Fluent builder pattern for configuration (AbrisConfig).
• SPI (Java ServiceLoader) for pluggable schema converters.
• Abstraction over Confluent registry (mockable for testing).
• Clear separation between “simple” Avro (static schema) and “Confluent” Avro (dynamic registry).
d. Key technologies / frameworks
• Apache Spark SQL (2.4+ / 3.x) and Spark’s DataFrame/Catalyst engine
• Apache Avro (binary serialization, schema definitions)
• Confluent Schema Registry (REST client)
• Scala (2.11/2.12/2.13), Maven, Java SPI, Log4j