question Re: create table as select failed, but only when stored as parquet in Support Questions

create table as select failed, but only when stored as parquet

vladenache — Mon, 31 Jan 2022 13:03:48 GMT

Hello,

A simple create table as select fails for some tables in my original_db and I don't know why.

The query I use is below, and the error HIVE gives out is really long, but the main theme is "Vertex Failed", "Error while processing row", "No enum constant" - as you can see below.

However, when ran without "STORED AS PARQUET" option, the query runs fine.

Any ideas what I can try to maybe create the tables as compressed parquet?

Thanks

beeline -e "SET parquet.compression=SNAPPY; SET hive.exec.compress.output=true; SET parquet.output.codec=snappy; set hive.tez.container.size = 8192; DROP table if exists DB.table; CREATE table DB.table STORED AS parquet As SELECT * FROM ORIGINAL_DB.table;"

ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1640013186854_27946_1_00, diagnostics=[Task failed, taskId=task_1640013186854_27946_1_00_000004, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1640013186854_27946_1_00_000004_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [...] Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.schema.OriginalType.q5

Re: create table as select failed, but only when stored as parquet

tarak271 — Tue, 01 Feb 2022 06:20:27 GMT

Hi @vladenache

From the pasted stack trace, we could see that enum q5 does not exist. From the code base

org.apache.parquet.schema.OriginalType

please refer to https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/schema/OriginalType.java

So the problem could be with

Incompatible versions of Parquet/Hive in use, please let us know what versions of hive, parquet are in use or Hadoop distribution version also helps
Source data format issues, please share schema of source table and sample data would help
Please share full stack trace to understand more about code execution path while query is getting executed

Re: create table as select failed, but only when stored as parquet

vladenache — Tue, 01 Feb 2022 07:25:08 GMT

Hello @tarak271 , thanks for your answer

Here are some more details

- Hadoop 3.1.1.7.1.7.63-1

[ I can't manage to get to the parquet version currently in use ]

- the table comes initially from a sqoop job, so maybe some invalid data slipped thru. over 100 columns, so difficult to present them here ( can't attach a sample in my response ) - only 3 types : int, string, double

I thought also it might be an invalid value in one of the columns but because I am not returned a row number along with the error message it is impossible to see where that would be.

Re: create table as select failed, but only when stored as parquet

tarak271 — Wed, 02 Feb 2022 08:06:07 GMT

HI @vladenache

Could you please share next 50 lines after the below line

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.schema.OriginalType.q5

or full output of beeline console which includes full stack trace

Re: create table as select failed, but only when stored as parquet

vladenache — Wed, 02 Feb 2022 08:44:23 GMT

It appears several times in the log

Here are the first lines of the error, which include that exception also:

ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1640013186854_27946_1_00, diagnostics=[Task failed, taskId=task_1640013186854_27946_1_00_000004, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1640013186854_27946_1_00_000004_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92) ... 19 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.schema.OriginalType.q5 at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:829) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1004) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552) ... 20 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.schema.OriginalType.q5 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:872) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:823) ... 27 more Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.schema.OriginalType.q5 at java.lang.Enum.valueOf(Enum.java:238) at org.apache.parquet.schema.OriginalType.valueOf(OriginalType.java:24) at org.apache.parquet.schema.MessageTypeParser.addPrimitiveType(MessageTypeParser.java:182) at org.apache.parquet.schema.MessageTypeParser.addType(MessageTypeParser.java:113) at org.apache.parquet.schema.MessageTypeParser.addGroupTypeFields(MessageTypeParser.java:101) at org.apache.parquet.schema.MessageTypeParser.parse(MessageTypeParser.java:94) at org.apache.parquet.schema.MessageTypeParser.parseMessageType(MessageTypeParser.java:84) at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.getSchema(DataWritableWriteSupport.java:51) at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.init(DataWritableWriteSupport.java:57) at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:418) at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:380) at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:70) at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:137) at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:126) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:294) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:279) ... 29 more ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1640013186854_27946_1_00_000004_1:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92) ... 19 more

Also, here is the schema created

INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:80t_lab.time, type:string, comment:null), FieldSchema(name:80t_lab.trigger, type:int, comment:null), FieldSchema(name:80t_lab.singlstate, type:tinyint, comment:null), FieldSchema(name:80t_lab.temppi_usecfc, type:double, comment:null), FieldSchema(name:80t_lab.humipi_usecfd, type:double, comment:null), FieldSchema(name:80t_lab.locorshtempglo, type:double, comment:null), FieldSchema(name:80t_lab.supply_temp_unfiltered, type:double, comment:null), FieldSchema(name:80t_lab.temp_setpoint, type:double, comment:null), FieldSchema(name:80t_lab.fan_setpoint, type:double, comment:null), FieldSchema(name:80t_lab.temp_local, type:double, comment:null), FieldSchema(name:80t_lab.return_temp, type:double, comment:null), FieldSchema(name:80t_lab.ht_humi, type:double, comment:null), FieldSchema(name:80t_lab.ht_dewpcorr, type:double, comment:null), FieldSchema(name:80t_lab.fc_temp, type:double, comment:null), FieldSchema(name:80t_lab.bara_fan, type:tinyint, comment:null), FieldSchema(name:80t_lab.bara_cool, type:tinyint, comment:null), FieldSchema(name:80t_lab.bara_deh, type:tinyint, comment:null), FieldSchema(name:80t_lab.bara_heate, type:tinyint, comment:null), FieldSchema(name:80t_lab.bara_hum, type:tinyint, comment:null), FieldSchema(name:80t_lab.bara_fc, type:tinyint, comment:null), FieldSchema(name:80t_lab.c1a_onoff, type:tinyint, comment:null), FieldSchema(name:80t_lab.c1b_onoff, type:tinyint, comment:null), FieldSchema(name:80t_lab.c2a_onoff, type:tinyint, comment:null), FieldSchema(name:80t_lab.c2b_onoff, type:tinyint, comment:null), FieldSchema(name:80t_lab.c3a_onoff, type:tinyint, comment:null), FieldSchema(name:80t_lab.c3b_onoff, type:tinyint, comment:null), FieldSchema(name:80t_lab.c4a_onoff, type:tinyint, comment:null), FieldSchema(name:80t_lab.c4b_onoff, type:tinyint, comment:null), FieldSchema(name:80t_lab.manscrramp1, type:tinyint, comment:null), FieldSchema(name:80t_lab.manscrramp2, type:tinyint, comment:null), FieldSchema(name:80t_lab.manscrramp3, type:tinyint, comment:null), FieldSchema(name:80t_lab.manscrramp4, type:tinyint, comment:null), FieldSchema(name:80t_lab.sys_pi_temp, type:double, comment:null), FieldSchema(name:80t_lab.sys_pi_fan, type:double, comment:null), FieldSchema(name:80t_lab.sys_pi_humi, type:double, comment:null), FieldSchema(name:80t_lab.c1_scrolldiff, type:double, comment:null), FieldSchema(name:80t_lab.c2_scrolldiff, type:double, comment:null), FieldSchema(name:80t_lab.c3_scrolldiff, type:double, comment:null), FieldSchema(name:80t_lab.c4_scrolldiff, type:double, comment:null), FieldSchema(name:80t_lab.c1_disct, type:double, comment:null), FieldSchema(name:80t_lab.c2_disct, type:double, comment:null), FieldSchema(name:80t_lab.c3_disct, type:double, comment:null), FieldSchema(name:80t_lab.c4_disct, type:double, comment:null), FieldSchema(name:80t_lab.slp1_status, type:string, comment:null), FieldSchema(name:80t_lab.slp2_status, type:string, comment:null), FieldSchema(name:80t_lab.slp3_status, type:string, comment:null), FieldSchema(name:80t_lab.slp4_status, type:string, comment:null), FieldSchema(name:80t_lab.pb1_status, type:string, comment:null), FieldSchema(name:80t_lab.pb2_status, type:string, comment:null), FieldSchema(name:80t_lab.pb3_status, type:string, comment:null), FieldSchema(name:80t_lab.pb4_status, type:string, comment:null), FieldSchema(name:80t_lab.pb1opspeed, type:tinyint, comment:null), FieldSchema(name:80t_lab.pb2opspeed, type:tinyint, comment:null), FieldSchema(name:80t_lab.pb3opspeed, type:tinyint, comment:null), FieldSchema(name:80t_lab.pb4opspeed, type:tinyint, comment:null), FieldSchema(name:80t_lab.gc1sped_f1, type:tinyint, comment:null), FieldSchema(name:80t_lab.gc2sped_f1, type:tinyint, comment:null), FieldSchema(name:80t_lab.gc3sped_f1, type:tinyint, comment:null), FieldSchema(name:80t_lab.gc4sped_f1, type:tinyint, comment:null), FieldSchema(name:80t_lab.leadlag_swap, type:string, comment:null), FieldSchema(name:80t_lab.dxqs_status_glo, type:string, comment:null), FieldSchema(name:80t_lab.pb1_nextst, type:string, comment:null), FieldSchema(name:80t_lab.pb2_nextst, type:string, comment:null), FieldSchema(name:80t_lab.pb3_nextst, type:string, comment:null), FieldSchema(name:80t_lab.pb4_nextst, type:string, comment:null), FieldSchema(name:80t_lab.pre1_pump_request_mode, type:string, comment:null), FieldSchema(name:80t_lab.pre1_pumpstartupstatus, type:tinyint, comment:null), FieldSchema(name:80t_lab.pre1_diff_press, type:double, comment:null), FieldSchema(name:80t_lab.pre2_pump_request_mode, type:string, comment:null), FieldSchema(name:80t_lab.pre2_pumpstartupstatus, type:tinyint, comment:null), FieldSchema(name:80t_lab.pre2_diff_press, type:double, comment:null), FieldSchema(name:80t_lab.pre3_diff_press, type:double, comment:null), FieldSchema(name:80t_lab.pre4_diff_press, type:double, comment:null), FieldSchema(name:80t_lab.pre_early_cmprtomixed, type:tinyint, comment:null), FieldSchema(name:80t_lab.pre_early_mixedtopump, type:tinyint, comment:null), FieldSchema(name:80t_lab.pre_early_cmprtopump, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb1_tempamb, type:double, comment:null), FieldSchema(name:80t_lab.gcb1_cntrlstate, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb11_cntrlssrc, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb12_cntrlssrc, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb11_compstate, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb12_compstate, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb1_fan1actual, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb1_fan2actual, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb1_fan3actual, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb1_fan1spdrqst, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb1_fan2spdrqst, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb1_fan3spdrqst, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcc1_temprfrg, type:double, comment:null), FieldSchema(name:80t_lab.gcc1_tempcondset, type:double, comment:null), FieldSchema(name:80t_lab.gcc1_prescond, type:double, comment:null), FieldSchema(name:80t_lab.gcc1_prescondset, type:double, comment:null), FieldSchema(name:80t_lab.gcb1_temprefrig, type:double, comment:null), FieldSchema(name:80t_lab.gcb1_prescond, type:double, comment:null), FieldSchema(name:80t_lab.gcb1_prescondset, type:double, comment:null), FieldSchema(name:80t_lab.gcb2_tempamb, type:double, comment:null), FieldSchema(name:80t_lab.gcb2_cntrlstate, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb21_cntrlssrc, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb22_cntrlssrc, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb21_compstate, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb22_compstate, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb2_fan1actual, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb2_fan1spdrqst, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcc2_temprfrg, type:double, comment:null), FieldSchema(name:80t_lab.gcc2_tempcondset, type:double, comment:null), FieldSchema(name:80t_lab.gcc2_prescond, type:double, comment:null), FieldSchema(name:80t_lab.gcc2_prescondset, type:double, comment:null), FieldSchema(name:80t_lab.pre1_out_p, type:double, comment:null), FieldSchema(name:80t_lab.pre1_inlet_p, type:double, comment:null), FieldSchema(name:80t_lab.pre1_inlet_t, type:double, comment:null), FieldSchema(name:80t_lab.pre1_out_t, type:double, comment:null), FieldSchema(name:80t_lab.gcc4_temprfrg, type:double, comment:null), FieldSchema(name:80t_lab.gcc4_prescond, type:double, comment:null), FieldSchema(name:80t_lab.eev1_reqstate, type:string, comment:null), FieldSchema(name:80t_lab.eev1_actstate, type:string, comment:null), FieldSchema(name:80t_lab.eev1_error_state, type:string, comment:null), FieldSchema(name:80t_lab.eev1_vcm_error_state, type:string, comment:null), FieldSchema(name:80t_lab.eev1_vcm_sensorerror, type:string, comment:null), FieldSchema(name:80t_lab.eev1_vcm_systemalarm, type:string, comment:null), FieldSchema(name:80t_lab.eev1_vcm_systemstategrp, type:string, comment:null), FieldSchema(name:80t_lab.eev1_vcm_press_psig, type:double, comment:null), FieldSchema(name:80t_lab.eev1_vcm_supht_degf, type:double, comment:null), FieldSchema(name:80t_lab.eev1_vcm_sh_setpt_degf, type:double, comment:null), FieldSchema(name:80t_lab.eev1_vcm_valvpos, type:double, comment:null), FieldSchema(name:80t_lab.eev1_vcm_valvposreq, type:double, comment:null), FieldSchema(name:80t_lab.eev1_vcm_valvcntrlen, type:tinyint, comment:null), FieldSchema(name:80t_lab.eev1_vcm_diginput, type:string, comment:null), FieldSchema(name:80t_lab.eev1_vcm_digoutput, type:string, comment:null), FieldSchema(name:80t_lab.eev2_reqstate_byte_1, type:string, comment:null), FieldSchema(name:80t_lab.eev2_actstate_byte_2, type:string, comment:null), FieldSchema(name:80t_lab.eev2_error_state_byte_3, type:string, comment:null), FieldSchema(name:80t_lab.eev2_vcm_error_state_byte_4, type:string, comment:null), FieldSchema(name:80t_lab.eev2_vcm_sensorerror, type:string, comment:null), FieldSchema(name:80t_lab.eev2_vcm_systemalarm, type:string, comment:null), FieldSchema(name:80t_lab.eev2_vcm_systemstategrp, type:string, comment:null), FieldSchema(name:80t_lab.eev2_vcm_press_psig, type:double, comment:null), FieldSchema(name:80t_lab.eev2_vcm_supht_degf, type:double, comment:null), FieldSchema(name:80t_lab.eev2_vcm_sh_setpt_degf, type:double, comment:null), FieldSchema(name:80t_lab.eev2_vcm_valvpos, type:double, comment:null), FieldSchema(name:80t_lab.eev2_vcm_valvposreq, type:double, comment:null), FieldSchema(name:80t_lab.eev2_vcm_valvcntrlen, type:tinyint, comment:null), FieldSchema(name:80t_lab.eev2_vcm_diginput, type:string, comment:null), FieldSchema(name:80t_lab.eev2_vcm_digoutput, type:string, comment:null), FieldSchema(name:80t_lab.pre2_out_p, type:double, comment:null), FieldSchema(name:80t_lab.pre2_inlet_p, type:double, comment:null), FieldSchema(name:80t_lab.pre2_inlet_t, type:double, comment:null), FieldSchema(name:80t_lab.pre2_out_t, type:double, comment:null), FieldSchema(name:80t_lab.eev4_supht, type:double, comment:null), FieldSchema(name:80t_lab.eev4_valvpos, type:tinyint, comment:null), FieldSchema(name:80t_lab.anaoutact1, type:int, comment:null), FieldSchema(name:80t_lab.anaoutact2, type:int, comment:null), FieldSchema(name:80t_lab.anaoutact3, type:int, comment:null), FieldSchema(name:80t_lab.anaoutact4, type:int, comment:null), FieldSchema(name:80t_lab.anain0, type:int, comment:null), FieldSchema(name:80t_lab.anain1, type:int, comment:null), FieldSchema(name:80t_lab.anain2, type:int, comment:null), FieldSchema(name:80t_lab.anain3, type:int, comment:null), FieldSchema(name:80t_lab.anain4, type:int, comment:null), FieldSchema(name:80t_lab.anain5, type:int, comment:null), FieldSchema(name:80t_lab.anain6, type:int, comment:null), FieldSchema(name:80t_lab.anain7, type:int, comment:null), FieldSchema(name:80t_lab.digou1_ss_q01, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou2_ss_q02, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou3_ss_q03, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou9i_ssq04, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou4_ss_q06, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou15_ssq08, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou14_ssq09, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou13_ssq10, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou12_ssq12, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou11_ssq13, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou10_ssq14, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou16_ssq15, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou5_ss_q17, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou6_ss_q18, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou7a_ssk03, type:tinyint, comment:null), FieldSchema(name:80t_lab.digou0w_ssk11, type:tinyint, comment:null), FieldSchema(name:80t_lab.digin0_ss_u15, type:tinyint, comment:null), FieldSchema(name:80t_lab.digin1_ss_u16, type:tinyint, comment:null), FieldSchema(name:80t_lab.digin2_ss, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginp0_ssu17, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginp1_ssu18, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginp2_ssu19, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginp3_ssu20, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginp4_ssu21, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginp5_ssu22, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginp6_ssu23, type:tinyint, comment:null), FieldSchema(name:80t_lab.digin13_ssu24, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginau_ssu25, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginhu_ssu26, type:tinyint, comment:null), FieldSchema(name:80t_lab.diginp7_ssu27, type:tinyint, comment:null), FieldSchema(name:80t_lab.digin14_ssu28, type:tinyint, comment:null), FieldSchema(name:80t_lab.digin15_ssu29, type:tinyint, comment:null), FieldSchema(name:80t_lab.eev_pre_st_glo, type:string, comment:null), FieldSchema(name:80t_lab.pre1_speedlimitrel, type:tinyint, comment:null), FieldSchema(name:80t_lab.pre2_speedlimitrel, type:tinyint, comment:null), FieldSchema(name:80t_lab.peeoopen, type:tinyint, comment:null), FieldSchema(name:80t_lab.peeost1, type:string, comment:null), FieldSchema(name:80t_lab.peeost2, type:string, comment:null), FieldSchema(name:80t_lab.gcb1_vsdspeedmodemax, type:tinyint, comment:null), FieldSchema(name:80t_lab.gcb2_vsdspeedmodemax, type:tinyint, comment:null), FieldSchema(name:80t_lab.peeosof1, type:tinyint, comment:null), FieldSchema(name:80t_lab.peeosof2, type:tinyint, comment:null), FieldSchema(name:80t_lab.fan_glo (q5), type:tinyint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20220131074416_e8fd6276-d028-42d0-9556-6d35bc7e1714); Time taken: 1.01 seconds

Re: create table as select failed, but only when stored as parquet

tarak271 — Fri, 18 Feb 2022 08:59:01 GMT

@vladenache The issue seems to be with field

FieldSchema(name:80t_lab.fan_glo (q5), type:tinyint, comment:null)], properties:null)

Please check schema of the table for above field and correct it's name. From the attached stack trace it is trying to identify q5 as a data type and failing as it does not exist

Re: create table as select failed, but only when stored as parquet

vladenache — Fri, 18 Feb 2022 11:45:24 GMT

thanks for noticing this, i missed it!

the table is a result of a sqoop job from mysql, so Ii guess the column name had q5 in it and somehow it ended up defined as a type ?!