weird Java version detection in HostInspector

[#1]

Hi,

today I successfully upgraded our cluster to CM4.8.0 and in parallel to Java7 (1.7.0_45, Oracle7). Everything ran smoothly and as expected.

Except one thing: after the upgrade I started the HostInspector and it told me that all nodes are using still the old java version 1.6.0_36

HostInspector is just kidding, I thought, but nope....after restarting the whole cluster the HostInspector still reported the old version.

But ALL the services on the cluster nodes have been started using java-1.7.0_45. To achive this I set the JAVA_HOME property on each "service environment safety valve" and I verified it by executing "ps auxf  | grep java" on the nodes.

I was wondering why the HostInspector still reports a different java version, and, more important, how he detects the version. After adding the Java7 path to the file /usr/lib64/cmf/service/common/cloudera-config.sh everything was fine, HostInspector now shows the correct java version for all nodes.

Now my question:

why does the HostInspector just rely on a setting in the mentioned shell script to determine (and display) the java version the cluster services are running with ? Is this on purpose or is this a bug ?!?!

I'd expect that the HostInspector is checking the REAL USED jvm of the running Hadoop service processes and displays it.

It's a weak source of truth if it reports a different version than the output of the process list tells.

What do you think?

many thanks...Gerd...

[#2]

This typically happens when the JDK is upgraded because the old JDK is still on the path and gets discovered first by the scripts that are looking for valid JDK versions.

Did you try these instructions to set your JAVA HOME in CM?

[#3]

Hi clint,

thanks for your response.

Yes, the link you provided is exactly the "how-to" that I followed, and at the end everything is working fine.

What I am wondering about is the fact that the HostInspector isn't determining the used java version from the running processes directly. Just relying on the first JDK found (from /usr/lib64/cmf/service/common/cloudera-config.sh) can definitely lead to wrong results, I mean the cluster is running and working fine on Java7 but the HostInspector tells the nodes are using Java6 just because the new path hasn't been added to this script,  isn't that weird ?

[#4 · Accepted solution]

Hi ge-ko,

Cloudera Manager generally has a strategy to pick which Java version to use.  The Host Inspector is intended for the common case: people only have one Java installed, and we want to make sure that CM can find good, consistent versions of Java.  If you want to dig into the details, /usr/lib64/cmf/service/common/cloudera-config.sh and /usr/lib/bigtop-utils/bigtop-detect-javahome might be of interest.

If you're customizing JAVA_HOME, you have two options: you can do it service-by-service with safety valves.  That works, but Host Inspector won't know about it.  That's ok.  You can also edit JAVA_HOME by editing /etc/default/cloudera-scm-agent.  See https://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Insta... for more details.

-- Philip

[#5]

Hi Philip,

many thanks for your additional explanation, sounds absolutely reasonable.

Keep up your great work for CM, brilliant tool 

best regards...Gerd...