Member since
01-30-2017
2
Posts
1
Kudos Received
0
Solutions
02-23-2017
08:43 AM
1 Kudo
Spark Thriftserver is unable to run a HiveUDTF. It throws the error that it is unable to find the functions although the function registration succeeds and the funtions does show up in the list output by show functions.
I am using a Hive UDTF, registering it using a jar placed on my local machine. Calling it using the following commands: //Registering the functions, this command succeeds. CREATE FUNCTION SampleUDTF AS 'com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF' USING JAR '/root/spark_files/experiments-1.2.jar'; //Thriftserver is able to look up the function, on this command: DESCRIBE FUNCTION SampleUDTF; Output:
+-----------------------------------------------------------+--+
| function_desc |
+-----------------------------------------------------------+--+
| Function: default.SampleUDTF |
| Class: com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF |
| Usage: N/A. |
+-----------------------------------------------------------+--+ // Calling the function: SELECT SampleUDTF('Paris'); Output :
Error: org.apache.spark.sql.AnalysisException: Undefined function: 'SampleUDTF'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7 (state=,code=0) I have also tried with using a non-local (on hdfs) jar, but I get the same error. My environment: HDP 2.5 with spark 2.0.0
... View more
Labels:
- Labels:
-
Apache Spark