How to set default time zone/local time for SparkSql on Hue Sessions.

[#1]

Hello

I am struggling to set the time zone for the sessions on Spark Sql while submitting the query over the HUE.Maybe somebody already has this problem and know how to solve it - I would be grateful for the help.

So far I checked that:

on pyspak session running on linux the time zone is the one i want :

spark.conf.get("spark.sql.session.timeZone")
'Europe/Warsaw'

However then i submit the TIMESTAMP on HUE running query against SparkSql I got the UTC Time Stamp.

SELECT TIMESTAMP 'now';

  TIMESTAMP '2024-07-02 08:01:06.157849' My current time

1	2024-07-02T06:01:06Z

 the UTC TIME

I have also added the variable to the spark-defaults.conf file like spark.sql.session.timeZone=Europe/Warsaw

But after restarting Spark3 and Hue nothing has changed.

Thank you for your help in advance.

Regards,

Bart

[#2 · Accepted solution]

Issue looks like you are hitting CDPD-66940 jira and try to upgrade your CDS3 parcel to latest and it will resolve your issue.

[#3 · Accepted solution]

@Bartlomiej Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

Thanks.

[#4]

Hello @Bartlomiej,

You can change the timezone for Hue using the steps below:

1. Log in to Clouder Manager as an Administrator.
2. Go to Clusters > Hue > Configuration > Time Zone and set a desired value.
   The default value is America/Los_Angeles. You can refer to https://timezonedb.com/time-zones for information on the timezones.
3. Click Save Changes.
4. Restart the Hue service.

Let me know if this helps.

Cheers!

[#5]

Hello 

Thank you for this hint. However I have already check that configuration and it was already  Europe/Warsaw so the one diesired. 

There must be  somewhere some other place to specify this i think.

[#6]

I think teh time zone has nothing to do with CURRENT_TIMESTAMP () finally. As teh actually need is to use the CURRENT_TIMESTAMP () or TIMESTAMP 'now' and have the local time as default time of that function in Spark SQL  while running the query over the HUE.

Now these function are giving the time 2 h late with means in UTC format and I would like to have by default 2h later or EUROPE/Berlin time for example

[#7 · Accepted solution]

Issue looks like you are hitting CDPD-66940 jira and try to upgrade your CDS3 parcel to latest and it will resolve your issue.

[#8 · Accepted solution]

@Bartlomiej Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

Thanks.

[#9 · Accepted solution]

Yes , upgrading spark to newest SPARK version SPARK3-3.3.2.3.3.7190.5-2-1.p0.54391297 - fixed the issue