How rs-works/hive-udfs works
Project Type and Purpose
• This is a Java-based Hive UDF (User Defined Function) library.
• Its purpose is to package a set of custom Hive functions (yearweek, yearweekdiff, yearmonth, yearmonthdiff, fetch_min, fetch_max, map_sum, geoip, urldecode) into a single JAR that can be loaded into Hive (or another Hadoop-compatible SQL engine) at runtime.
File Structure Analysis
• Top level:
• build.gradle, gradlew/gradlew.bat, settings.gradle – Gradle build and wrapper scripts.
• .circleci/config.yml – CI pipeline definition.
• README.md – documentation of each UDF.
• src/main/java/rsworks/hive – contains one Java class per Hive UDF.
• src/main/resources – placeholder for any runtime resources (e.g., GeoIP database file).
• src/test/java/rsworks/hive – unit tests covering key UDFs.
README Insights
• Describes each function’s signature, use case examples, and edge-cases.
Components
a. Core Components
1. UDF Library Module (Gradle project)
– Source code (src/main/java/rsworks/hive/*.java)
– Unit tests (src/test/java/rsworks/hive)
– Build artifacts: hive-udfs.jar
2. Build & CI Pipeline
– Gradle (compile, test, jar)
– CircleCI (checkout → build → test → publish)
3. Hive Execution Environment
– HiveServer2 or Hive CLI
– ClassLoader / ADD JAR mechanism
– GeoIP Database file (external resource loaded with ADD FILE)
4. End-User / Client
– SQL clients (Beeline, Spark SQL, etc.)
– Queries invoking the custom UDFs
b. Relationships & Data Flows
1. Developer commits code → CircleCI pulls repository → Gradle build + unit tests → produces hive-udfs.jar
2. Admin or user copies hive-udfs.jar (and GeoIP MMDB) to HDFS or local path
3. In Hive CLI/Beeline:
• JAR hive-udfs.jar
• FILE GeoLite2-City.mmdb
• CREATE TEMPORARY FUNCTION <fn> AS 'rsworks.hive.<ClassName>'
4. Hive execution engine loads the class from the JAR, invokes UDF methods at query runtime
5. geoip UDF may read the MMDB file via the resource path
c. Architectural Patterns & Principles
• Plugin architecture: Each UDF is a standalone Java class implementing Hive’s UDF interface, discovered at runtime via CREATE FUNCTION.
• Build/test automation: Gradle for dependency management & compilation, JUnit for unit tests, CircleCI for continuous integration.
• Separation of concerns:
• Source files under src/main/java
• Tests under src/test/java
• Build definitions at root
d. Key Technologies & Frameworks
• Java 8+
• Apache Hive UDF API
• Gradle
• JUnit (unit testing)
• CircleCI (CI/CD)
• GeoLite2 (external binary resource)