Member since
03-04-2017
1
Post
1
Kudos Received
0
Solutions
03-04-2017
01:22 AM
1 Kudo
Hey I have the same problem here! With spark-1.6.2-bin-hadoop2.6 and spark-2.0.1-bin-hadoop2.6 I get the following behavior: spark-sql> describe function tc.dt_to_date;Function: tc.dt_to_date
Class: com.XXXX.DTToDateUsage: N/A.
Time taken: 0.127 seconds, Fetched 3 row(s)
spark-sql> select tc.dt_to_date('2016-11-01') from dwh.dim_geography limit 1;
17/03/04 01:26:49 INFO execution.SparkSqlParser: Parsing command: select tc.dt_to_date('2016-11-01') from dwh.dim_geography limit 1
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: int
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: string
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: string
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: int
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: string
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: int
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: string
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: int
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: int
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: string
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: string
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: int
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: string
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: int
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: string
17/03/04 01:26:49 INFO parser.CatalystSqlParser: Parsing command: int
Error in query: Undefined function: 'tc.dt_to_date'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7
I used the --jars option to initialize spark-sql com command line and referred to the jar package where the function is defined. Notice how describe function is able to identify the pacakge name but the Usage field remains as N/A. With spark-1.5.0-bin-hadoop2.6 and spark-1.5.2-bin-hadoop2.6 it works fine.
... View more