Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

custom jars in spark sql is failing with Error in query: undefined function

custom jars in spark sql is failing with Error in query: undefined function

Guru

I am getting following error when using custom jars in spark-sql, so can someone please help me to solve it.

17/09/05 05:31:02 INFO ParseDriver: Parse Completed

Error in query: undefined function test_db.COLLECT; line 1 pos 59

spark-sql> add jar /home/root/brickhouse-0.6.0.jar;

ADD JAR /home/root/brickhouse-0.6.0.jar

Added [/home/root/brickhouse-0.6.0.jar] to class path 17/09/05 06:01:24 INFO SessionState:

Added [/home/root/brickhouse-0.6.0.jar] to class path

Added resources: [/home/root/brickhouse-0.6.0.jar]

17/09/05 06:01:24 INFO SessionState: Added resources: [/home/root/brickhouse-0.6.0.jar]

ADD JAR /home/s0998dnz/brickhouse-0.6.0.jar Added [/home/root/brickhouse-0.6.0.jar] to class path

use test_db;

select category, test_db.COLLECT(uom_name_code, attr_list) as uom_map FROM (SELECT category,uom_name_code, collect_list(attribute_name) as attr_list FROM sample WHERE ((uom_name_code is not null) or (length(trim(uom_name_code)) > 0) ) AND lower(response_type) in ('number only' , 'drop-down uom') GROUP BY category,uom_name_code) a GROUP BY category limit 3;

17/09/05 05:31:02 INFO ParseDriver: Parse Completed

Error in query: undefined function test_db.COLLECT; line 1 pos 59


3 REPLIES 3

Re: custom jars in spark sql is failing with Error in query: undefined function

Expert Contributor

Hi, @Saurabh

Did you do `CREATE FUNCTION`?

Re: custom jars in spark sql is failing with Error in query: undefined function

Guru
@Dongjoon Hyun

Can we use function in spark-sql which was created parmanent function in hive ?

Highlighted

Re: custom jars in spark sql is failing with Error in query: undefined function

New Contributor

It´s not working to me using permanent function as well, just received the message "Error in query: undefined function", but the function were create in Hive., when I checked using "show function my_database" I can see the function there.

Don't have an account?
Coming from Hortonworks? Activate your account here