Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issue while querying orc partitioned table through spark-sql

Issue while querying orc partitioned table through spark-sql

New Contributor

Hi all,

I am currently working on HDP 2.4 with Spark 1.6.1 and Hive 1.2.1.

After inserting partition into an orc partitioned table (zlib compressed) using spark, i'm not able to query that table using spark sql.

Here is the error i get :

org.apache.spark.sql.catalyst.util.DataTypeException: Unsupported dataType: struct<filed1:string,filed2:string,field:string,process_date:date>. If you have a struct and a field name of it has any special characters, please use backticks (`) to quote that field name, e.g. `x+y`. Please note that backtick itself is not supported in a field name.

Note that the table is partitioned on another date field that doesn't appear in the error

Using hive, this table is fully usable.

Any help would be useful,

Thanks.

3 REPLIES 3

Re: Issue while querying orc partitioned table through spark-sql

Guru

Can you please copy and paste the DDL of the ORC table? Also, what do you mean by inserting the partition using Spark? Are you running a SparkSQL command to specify the partition on inserting the data? Is this an external or an internal table?

Re: Issue while querying orc partitioned table through spark-sql

New Contributor

Hi Vedant,

By inserting the partition using spark i mean to use the following command :

df.write.mode("append").partitionBy("col6").saveAsTable(target_table)

I'm working with external table as you can see here :

CREATE EXTERNAL TABLE database.table (  
  key_1 string,  
  key_2 string,  
  col3 string,  
  col4 double,   
  col5 date)
PARTITIONED BY (col6 date)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\073'
LINES TERMINATED BY '\n'
STORED AS ORC
LOCATION'/user/jdoe/data/table'
TBLPROPERTIES ('orc.compress'='ZLIB');

Thanks for your help.

Re: Issue while querying orc partitioned table through spark-sql

New Contributor

Hi !

No idea anyone ?

Thx.