Skip to main content
gbraccialli3
gbraccialli3
Gold ⭐️
October 2, 2015

Using Hive UDF/UDAF/UDTF with SparkSQL

  • October 5, 2026 at 9:22 AM
  • 14 replies
  • 26072 views
  • Translate All

Sharing the steps to make Hive UDF/UDAF/UDTF to work natively with SparkSQL

1- Open spark-shell with hive udf jar as parameter:

spark-shell --jars path-to-your-hive-udf.jar

2- From spark-shell, open declare hive context and create functions

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc);

sqlContext.sql("""create temporary function balance as 'com.github.gbraccialli.hive.udf.BalanceFromRechargesAndOrders'""");


3- From spark-shell, use your UDFs directly in SparkSQL:

sqlContext.sql("""
create table recharges_with_balance_array as
select
reseller_id,
phone_number,
phone_credit_id,
date_recharge,
phone_credit_value,
balance(orders,'date_order', 'order_value', reseller_id, date_recharge, phone_credit_value) as balance
from orders
""");

PS: I found some issues using UDTFs with spark 1.3, which was fixed on spark 1.4.1. I tested all, UDF, UDAF and UDTF, all of them worked properly, same sql statements and same results.

    14 replies

    azeltov
    azeltov
    Gold ⭐️
    November 11, 2015

    @Guilherme Braccialli thanks for the write up. Would love to see a sample implementation of the actual UDF. A basic shell of creating a scala project for the udf.

    Translate
    gbraccialli3
    gbraccialli3Author
    Gold ⭐️
    November 11, 2015

    @azeltov@hortonworks.com see below github project for hive udf, exactly same jar built for hive project works fine with spark. The scala code is above:

    https://github.com/gbraccialli/HiveUtils

    Translate
      azeltov
      azeltov
      Gold ⭐️
      November 11, 2015

      Awesome! Will check it out. Another question, i had problems with zeppelin registering a custom jar, i had to use a maven dependency, could not load it using the path. It works fine using CLI and passing --jars path-to-your-hive-udf.jar argument. Curious if u tried it in zeppelin?

      Translate
      azeltov
      azeltov
      Gold ⭐️
      November 11, 2015
      @Guilherme Braccialli

      Hopefully that JIRA will be fixed soon. Did you register the custom jar in the local maven of your sandbox? And that still did not work?

      Translate
      gbraccialli3
      gbraccialli3Author
      Gold ⭐️
      November 11, 2015

      @azeltov@hortonworks.com

      See what I tried.

      First, with spark-shell

      1- Download jar and register to local maven

      su - zeppelin

      wget https://raw.githubusercontent.com/gbraccialli/HiveUtils/master/target/HiveUtils-1.0-SNAPSHOT-jar-with-dependencies.jar /tmp/HiveUtils-1.0-SNAPSHOT-jar-with-dependencies.jar

      mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file \
      -Dfile=/tmp/HiveUtils-1.0-SNAPSHOT-jar-with-dependencies.jar \
      -DgroupId=com.github.gbraccialli \
      -DartifactId=HiveUtils \
      -Dversion=1.0-SNAPSHOT \
      -Dpackaging=jar

      Translate
        gbraccialli3
        gbraccialli3Author
        Gold ⭐️
        November 11, 2015

        2- Run spark-shell with dependency

        spark-shell --master yarn-client --packages "com.github.gbraccialli:HiveUtils:1.0-SNAPSHOT"

        3- Run spark code

        val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc);
        sqlContext.sql("""create temporary function geohash_encode as 'com.github.gbraccialli.hive.udf.UDFGeohashEncode'""");
        sqlContext.sql("""select geohash_encode(1.11,1.11,3) from sample_07 limit 10""").collect().foreach(println);

        spark-shell worked fine!

        Translate
        zhanghn
        zhanghn
        Silver ⭐️
        April 8, 2016

        well , i tried create temporary function in beeline , and failed ; and create function, the function was created , and can be desc function. but , when i accessing it , it show can not find the function . so , do you have the same problem. I'm trying to let everyone connected to my thriftserver to have access to udf that I deployed. do you have any suggestions?

        Translate
        dz186001
        dz186001
        Silver ⭐️⭐️⭐️
        May 26, 2017

        Hi @Guilherme Braccialli, so you did not run into this issue?

        https://issues.apache.org/jira/browse/SPARK-20033

        Thank you.

        Translate
        Terms & ConditionsAccessibility statement