Error while inserting the data to another HIVE tables created in parquet format

[#1]

Hello Experts !

I have create the table1 using the custom cobol serde and table 2 with the same ddl but the storage format is different(parquet) and partitioned , later while trying to insert the data into table2 using the fololwing statement, I am getting the below error. Could you please have a look and guide me?

Statement:

insert overwrite table table2 PARTITION(dt='2016.06.13') select * from table1;

.

Error:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"c5qtsys_detail_clnt_no":"2430","c5qtsys_detail_acct_no":"4510109900005012 ","c5qtsys_detail_log_date":20160406,"c5qtsys_detail_auth_log_time":151837,"c5qtsys_detail_maint_date":20151020,"c5qtsys_detail_maint_time":131009,"c5qtsys_detail_operator_id":"BLANMA","c5qtsys_detail_prev_frd_flag":"N","c5qtsys_detail_curr_frd_flag":"N","c5qtsys_detail_trans_amt":"+0000003.61","c5qtsys_detail_message_type":100,"c5qtsys_detail_reference_no":" ","c5qtsys_detail_trans_date":2015289,"c5qtsys_detail_trans_id":"305289672360311 ","c5qtsys_detail_trans_apvl_cd":"010837","c5qtsys_detail_merchant_class":"8661","filler":" "} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.hive.serde2.io.ParquetHiveRecord at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:753) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)

[#2 · Accepted solution]

@Venkat Chinnari

The issue seems to be with cast from text to parquet. Try creating a sample table say table3 without serde properties but just 'stored as parquet' and check if insert overwrite works.

Thanks and Regards,

Sindhu

[#3]

Thanks Sindhu,

I have already tested that and working fine. However, I am wonder why it is not working.

[#4]

can you post the SQL and table definition?

You may have incorrect properties as Parquet is not the same as a CSV file.

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC

[#5]

CREATE EXTERNAL TABLE `table1 `( | | columns ) | | PARTITIONED BY ( | | `dt` string) | | ROW FORMAT DELIMITED | | FIELDS TERMINATED BY '|' | | STORED AS INPUTFORMAT | | 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' | | OUTPUTFORMAT | | 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' | | LOCATION | | 'hdfs:location' | | TBLPROPERTIES ( | | 'transient_lastDdlTime'='1465914063')

[#6]

parquet isnotcomma delimited

[#7]

Got the same issue. Run hive command "desc formatted TABLE_NAME" found :

# Storage Information

SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat

OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat

The error is caused by SerDe Library.

Drop table & recreate table, now the SerDe Library is org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe .

Now it's working.