I am currently working on HDP 2.4 with Spark 1.6.1 and Hive 1.2.1.
After inserting partition into an orc partitioned table (zlib compressed) using spark, i'm not able to query that table using spark sql.
Here is the error i get :
org.apache.spark.sql.catalyst.util.DataTypeException: Unsupported dataType: struct<filed1:string,filed2:string,field:string,process_date:date>. If you have a struct and a field name of it has any special characters, please use backticks (`) to quote that field name, e.g. `x+y`. Please note that backtick itself is not supported in a field name.
Note that the table is partitioned on another date field that doesn't appear in the error
Using hive, this table is fully usable.
Any help would be useful,
Can you please copy and paste the DDL of the ORC table? Also, what do you mean by inserting the partition using Spark? Are you running a SparkSQL command to specify the partition on inserting the data? Is this an external or an internal table?
By inserting the partition using spark i mean to use the following command :
I'm working with external table as you can see here :
CREATE EXTERNAL TABLE database.table ( key_1 string, key_2 string, col3 string, col4 double, col5 date) PARTITIONED BY (col6 date) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\073' LINES TERMINATED BY '\n' STORED AS ORC LOCATION'/user/jdoe/data/table' TBLPROPERTIES ('orc.compress'='ZLIB');
Thanks for your help.