Member since
03-04-2016
1
Post
0
Kudos Received
0
Solutions
01-05-2021
02:42 AM
Hi @GopiG, There are several issues, you have to consider. 1. How has the data been written to the Kudu table. - via Impala: timestamp remains local - via Spark: timestamp will be converted to UTC in Kudu (however you can change this behavior in spark.conf) 2. Reading the Kudu table in Spark Timestamp will be converted from UTC to local, so you have local times in your data frame. 3. Writing the DataFrame to Hive parquet Local timestamp is converted to UTC. You have to check the following configuration options: - spark.sql.parquet.int96TimestampConversion - use_local_tz_for_unix_timestamp_conversions - convert_legacy_hive_parquet_utc_timestamps https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_timestamp.html
... View more