Parquet Decimal Datatype Returning an Error in Impala
Hi,
We are having an issue reading tables in Impala with datatype Decimal (1,1). The error is:
column 'DECIMAL_1_1' has an invalid type length. Expecting: 1 len in file 4
We have two Cloudera clusters (CDH 6.3.3). This error only occurs in the 2nd cluster.
We write data into Parquet files through IBM Datastage.
I run the same job in both clusters to examine the parquet files schema using parquet-tools schema , and I found this:
In the 1st cluster, in the parquet file the schema of the column causing the issue is:
optional int32 DECIMAL_1_1 (DECIMAL(1,1));
However, in the 2nd cluster:
optional fixed_len_byte_array(4) DECIMAL_1_1 (DECIMAL(1,1));
We need to understand why the type is fixed_len_byte_array(4) in this parquet while it's int32 in the other. Apparently the fixed_len_byte_array(4) is causing the issue with Impala.

