question Re: Error while inserting the data to another HIVE tables created in parquet format in Archives of Support Questions (Read Only)

Error while inserting the data to another HIVE tables created in parquet format

venkatibrahmam_ — Tue, 14 Jun 2016 21:06:25 GMT

Hello Experts !

I have create the table1 using the custom cobol serde and table 2 with the same ddl but the storage format is different(parquet) and partitioned , later while trying to insert the data into table2 using the fololwing statement, I am getting the below error. Could you please have a look and guide me?

Statement:

insert overwrite table table2 PARTITION(dt='2016.06.13') select * from table1;

Error:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"c5qtsys_detail_clnt_no":"2430","c5qtsys_detail_acct_no":"4510109900005012 ","c5qtsys_detail_log_date":20160406,"c5qtsys_detail_auth_log_time":151837,"c5qtsys_detail_maint_date":20151020,"c5qtsys_detail_maint_time":131009,"c5qtsys_detail_operator_id":"BLANMA","c5qtsys_detail_prev_frd_flag":"N","c5qtsys_detail_curr_frd_flag":"N","c5qtsys_detail_trans_amt":"+0000003.61","c5qtsys_detail_message_type":100,"c5qtsys_detail_reference_no":" ","c5qtsys_detail_trans_date":2015289,"c5qtsys_detail_trans_id":"305289672360311 ","c5qtsys_detail_trans_apvl_cd":"010837","c5qtsys_detail_merchant_class":"8661","filler":" "} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.hive.serde2.io.ParquetHiveRecord at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:753) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)

Re: Error while inserting the data to another HIVE tables created in parquet format

ssubhas — Tue, 14 Jun 2016 21:14:48 GMT

@Venkat Chinnari

The issue seems to be with cast from text to parquet. Try creating a sample table say table3 without serde properties but just 'stored as parquet' and check if insert overwrite works.

Thanks and Regards,

Sindhu

Re: Error while inserting the data to another HIVE tables created in parquet format

TimothySpann — Tue, 14 Jun 2016 21:26:26 GMT

can you post the SQL and table definition?

You may have incorrect properties as Parquet is not the same as a CSV file.

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC

Re: Error while inserting the data to another HIVE tables created in parquet format

venkatibrahmam_ — Tue, 14 Jun 2016 21:30:25 GMT

Thanks Sindhu,

I have already tested that and working fine. However, I am wonder why it is not working.

Re: Error while inserting the data to another HIVE tables created in parquet format

venkatibrahmam_ — Tue, 14 Jun 2016 21:32:07 GMT

Re: Error while inserting the data to another HIVE tables created in parquet format

TimothySpann — Tue, 05 Jul 2016 06:20:24 GMT

parquet isnotcomma delimited

Re: Error while inserting the data to another HIVE tables created in parquet format

awesome — Fri, 25 Jan 2019 22:10:49 GMT

Got the same issue. Run hive command "desc formatted TABLE_NAME" found :

# Storage Information

SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat

OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat

The error is caused by SerDe Library.

Drop table & recreate table, now the SerDe Library is org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe .

Now it's working.