【Problem】
After upgrading Cloudera Manager to 7.11.3, attempting to reboot the cluster causes an error in the Lily HBase Indexer of the Key-Value Store Indexer.
【Use】
CDP Private Cloud Base 7.1.8 (Will upgrade to 7.1.9 later)
OS: RHEL 7.6
Java: OpenJDK 11.0.8
Python: 3.8.12
【Occurrence procedure】
Before upgrading, I installed Python 3.8 on all hosts that contained Cloudera Manager Agents.
The installation was done according to the following documentation:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/installation/topics/cdpdc-cm-install-python-3...
Then, according to the following documentation, Cloudera Manager was upgraded from 7.7.1 to 7.11.3.
https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdp/topics/ug_cm_upgrade.html
Verified that the Server and Agent were upgraded successfully.
Therefore, I proceeded with the steps in Step 5: After You Upgrade, "Perform Post-Upgrade Steps".
When I rebooted the cluster in step 4, I encountered an error with the Lily HBase Indexer of the Key-Value Store Indexer.
[Error details]
stderr had the following output:
---------
+ HBASE_INDEXER_OPTS='-Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh - Dsun.security.krb5.disableReferrals=true -Djdk.tls.ephemeralDHKeySize=2048 -Dcom.sun.management.jmxremote.ssl.enabled.protocols=TLSv1.2,TLSv1.3'
+ '[' jaas.conf '!=' '' ']'
+ export 'HBASE_INDEXER_OPTS=-Djava.security.auth.login.config=/var/run/cloudera-scm-agent/process/1546484214-ks_indexer-HBASE_INDEXER/jaas.conf - Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh - Dsun.security.krb5.disableReferrals=true -Djdk.tls.ephemeralDHKeySize=2048 -Dcom.sun.management.jmxremote.ssl.enabled.protocols=TLSv1.2,TLSv1.3'
+ HBASE_INDEXER_OPTS='-Djava.security.auth.login.config=/var/run/cloudera-scm-agent/process/1546484214-ks_indexer-HBASE_INDEXER/jaas.conf -Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -Dsun.security.krb5.disableReferrals=true - Djdk.tls.ephemeralDHKeySize=2048 -Dcom.sun.management.jmxremote.ssl.enabled.protocols=TLSv1.2,TLSv1.3'
+ export HBASE_INDEXER_CONF_DIR=/var/run/cloudera-scm-agent/process/1546484214-ks_indexer-HBASE_INDEXER
+ HBASE_INDEXER_CONF_DIR=/var/run/cloudera-scm-agent/process/1546484214-ks_indexer-HBASE_INDEXER
+ exec /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/hbase-solr/bin/hbase-indexer server
File "<string>", line 1
import os; import sys; print os.path.realpath(sys.argv[1])
^
SyntaxError: invalid syntax
/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/hbase-solr/bin/hbase-indexer: line 56: /run/cloudera-scm-agent/process/1546484214-ks_indexer-HBASE_INDEXER/ hbase-indexer-config.sh: No such file or directory
Error: Could not find or load main class com.ngdata.hbaseindexer.Main
Caused by: java.lang.ClassNotFoundException: com.ngdata.hbaseindexer.Main
----------
After "Reboot Cluster", all services started before this service are running normally.
Also, all hosts in the cluster are running normally.
How do you deal with this problem?
