Hello cloudera community.
I have a timezone problem which seems to be similar to this:
Since about a month or longer I have no more impala queries in the query monitor section, which makes it hard to optimize statements
The log of the management host has a lot of entries like that:
tail -f /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-`hostname -f`.log.out
2019-11-19 18:25:43,536 WARN com.cloudera.cmon.firehose.ImpalaQueryManager: Impala query: c8476c1bdfa67dac:d50e6e5b940d6bba with endTime 2019-11-19T19:25:41.826Z is outside acceptance window.
It looks like the log timestamp is at least an hour past the query endtime timestamp
server time was
Di 19. Nov 19:18:25 CET 2019
I wrote a Java snippet to get the Java system time
# java TimeTest
Current time in milliseconds = 1574123366996 => Tue Nov 19 01:29:26 MEZ 2019
Current time zone: Europe/Berlin
Di 19. Nov 01:29:26 CET 2019
Which looks good to me.
In the above mentioned post the solution was:
"After further investigation, I found out that the timezone "seen" by Java was different than in OS. I added this startup parameter to the service monitor configuration which configures the timezone
The question is: if I want to add teh needed timezon -Duser.timezone=Europe/Berlin where should I configure it?
We tried in /etc/default/cloudera-scm-agent.
But after a restart of server & manager there was noc change
Still the same problem
Would appreciate any help!
There is perhaps a difference in localtime configured in your impala daemon nodes and management service nodes which is causing these issues.
You can change the system localtime on the impala daemons hosts using below steps:
1) delete the /etc/localtime file. 2) Link the time CET the /usr/share/zoneinfo folder to /etc/localtime with this command:
#ln -s /usr/share/zoneinfo/CET /etc/localtime
2.a) If you don't want to link it you can copy and rename it to localtime:
#cp /usr/share/zoneinfo/CET /etc/localtime
After you did this on the hosts where the timezone is wrong, please execute the queries and check if there are still discrepancies observed
An easier approach would be to change timezone of your service monitor host/CM server. You can follow the same steps to have the same timezone as the Impala daemon nodes. Restart the Cm server for changes to take effect on the CM web UI
I had the same problem than you.
I also read @Andrej solution, and as you, I didn't know where to put that...
Then, I read your post I saw you put it in /etc/default/cloudera-scm-agent without good results.
So, I try it on /etc/default/cloudera-scm-server and worked... one of the last lines is CMF_JAVA_OPTS=, so I just put it at the end of it, like this:
CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Duser.timezone=Americas/Santiago"