Hi clouder people, I am really surprised reading this documentation around parquet files & timestamp type & hive VS impala: https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_timestamp.html. Above all for this two things: When the table uses Parquet format, Impala expects any time zone adjustment to be applied prior to writing, while TIMESTAMP values written by Hive are adjusted to be in the UTC time zone. When Hive queries Parquet data files that it wrote, it adjusts the TIMESTAMP values back to the local time zone, while Impala does no conversion. Hive does no time zone conversion when it queries Impala-written Parquet files. If you have data files written by Hive, those TIMESTAMP values represent the local timezone of the host where the data was written, potentially leading to inconsistent results when processed by Impala. To avoid compatibility problems or having to code workarounds, you can specify one or both of these impaladstartup flags: -use_local_tz_for_unix_timestamp_conversions=true -convert_legacy_hive_parquet_utc_timestamps=true. Although -convert_legacy_hive_parquet_utc_timestamps is turned off by default to avoid performance overhead, Cloudera recommends turning it on when processing TIMESTAMP columns in Parquet files written by Hive, to avoid unexpected behavior. I would like to ask your opinion considering the previous link about what is the best way to write a timestamp value (in a parquet file) throught HIVE or / and SPARK to be queried with HIVE and IMPALA and SPARK? Writing the file using HIVE or / and SPARK and suffering the derivated performance problem of setting this two properties -use_local_tz_for_unix_timestamp_conversions=true -convert_legacy_hive_parquet_utc_timestamps=true. Writing the file using IMPALA (preparing the table with HIVE or SPARK previously to avoid complex queries in Impala), due to: Hive does no time zone conversion when it queries Impala-written Parquet files. Another one? Regards,
... View more
