Support Questions

Find answers, ask questions, and share your expertise

custom jars in spark sql is failing with Error in query: undefined function

Guru

I am getting following error when using custom jars in spark-sql, so can someone please help me to solve it.

17/09/05 05:31:02 INFO ParseDriver: Parse Completed

Error in query: undefined function test_db.COLLECT; line 1 pos 59

spark-sql> add jar /home/root/brickhouse-0.6.0.jar;

ADD JAR /home/root/brickhouse-0.6.0.jar

Added [/home/root/brickhouse-0.6.0.jar] to class path 17/09/05 06:01:24 INFO SessionState:

Added [/home/root/brickhouse-0.6.0.jar] to class path

Added resources: [/home/root/brickhouse-0.6.0.jar]

17/09/05 06:01:24 INFO SessionState: Added resources: [/home/root/brickhouse-0.6.0.jar]

ADD JAR /home/s0998dnz/brickhouse-0.6.0.jar Added [/home/root/brickhouse-0.6.0.jar] to class path

use test_db;

select category, test_db.COLLECT(uom_name_code, attr_list) as uom_map FROM (SELECT category,uom_name_code, collect_list(attribute_name) as attr_list FROM sample WHERE ((uom_name_code is not null) or (length(trim(uom_name_code)) > 0) ) AND lower(response_type) in ('number only' , 'drop-down uom') GROUP BY category,uom_name_code) a GROUP BY category limit 3;

17/09/05 05:31:02 INFO ParseDriver: Parse Completed

Error in query: undefined function test_db.COLLECT; line 1 pos 59


3 REPLIES 3

Expert Contributor

Hi, @Saurabh

Did you do `CREATE FUNCTION`?

Guru
@Dongjoon Hyun

Can we use function in spark-sql which was created parmanent function in hive ?

New Contributor

It´s not working to me using permanent function as well, just received the message "Error in query: undefined function", but the function were create in Hive., when I checked using "show function my_database" I can see the function there.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.