Skip to main content
vladenache
vladenache
Silver ⭐️⭐️
January 31, 2022
Solved

create table as select failed, but only when stored as parquet

Hello,

 

A simple create table as select fails for some tables in my original_db and I don't know why.

The query I use is below, and the error HIVE gives out is really long, but the main theme is "Vertex Failed", "Error while processing row", "No enum constant" - as you can see below.

 

However, when ran without "STORED AS PARQUET" option, the query runs fine.

Any ideas what I can try to maybe create the tables as compressed parquet?

 

Thanks

 

 

 

beeline -e "SET parquet.compression=SNAPPY;
SET hive.exec.compress.output=true;
SET parquet.output.codec=snappy;
set hive.tez.container.size = 8192;


DROP table if exists DB.table;
CREATE table DB.table
STORED AS parquet

As


SELECT * FROM ORIGINAL_DB.table;"

 

ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1640013186854_27946_1_00, diagnostics=[Task failed, taskId=task_1640013186854_27946_1_00_000004, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1640013186854_27946_1_00_000004_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row

[...]

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.schema.OriginalType.q5

 

tarak271
tarak271Answer
Gold ⭐️
February 18, 2022

@vladenache The issue seems to be with field 

FieldSchema(name:80t_lab.fan_glo (q5), type:tinyint, comment:null)], properties:null)

Please check schema of the table for above field and correct it's name. From the attached stack trace it is trying to identify q5 as a data type and failing as it does not exist 

 

View in thread

6 replies

tarak271
Gold ⭐️
tarak271
Gold ⭐️
February 1, 2022

Hi @vladenache 

 

From the pasted stack trace, we could see that enum q5 does not exist. From the code base 

org.apache.parquet.schema.OriginalType

please refer to https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/schema/OriginalType.java

So the problem could be with 

  1. Incompatible versions of Parquet/Hive in use, please let us know what versions of hive, parquet are in use or Hadoop distribution version also helps
  2. Source data format issues, please share schema of source table and sample data would help
  3. Please share full stack trace to understand more about code execution path while query is getting executed
Translate
    vladenache
    vladenacheAuthor
    Silver ⭐️⭐️
    February 1, 2022

    Hello @tarak271 , thanks for your answer 

     

    Here are some more details

    - Hadoop 3.1.1.7.1.7.63-1

    [ I can't manage to get to the parquet version currently in use ]

    - the table comes initially from a sqoop job, so maybe some invalid data slipped thru. over 100 columns, so difficult to present them here ( can't attach a sample in my response ) - only 3 types : int, string, double

     

    I thought also it might be an invalid value in one of the columns but because I am not returned a row number along with the error message it is impossible to see where that would be.

     

     

     

     

    Translate
    tarak271
    Gold ⭐️
    tarak271
    Gold ⭐️
    February 2, 2022

    HI @vladenache 

     

    Could you please share next 50 lines after the below line

     

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.schema.OriginalType.q5

     

    or full output of beeline console which includes full stack trace

    Translate
    Terms & ConditionsAccessibility statement