question Re: After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase in Support Questions

After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

Debashish — Fri, 16 Sep 2022 14:31:07 GMT

Performed the following steps to disable Kerberos

Zookeeper -> enableSecurity (Enable Kerberos Authentication)-> false
HDFS -> hadoop.security.authentication -> Simple
HDFS -> hadoop.security.authorization -> false
HDFS -> dfs.datanode.address -> from 1004 (for Kerberos) to default
HDFS -> dfs.datanode.http.address -> from 1006 (for Kerberos) to default
HDFS -> Data Directory Permissions -> from 700 to 755
HBASE -> hbase.security.authentication -> Simple
HBASE -> hbase.security.authorization -> false

After doing so, all services started except Hbase.

Found this in error logs

Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied.

How to solve this problem?

Re: After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

bgooley — Thu, 18 Jul 2019 05:22:46 GMT

@Debashish,

The "supervisor_status" file is not used by hbase, so this message should be non-fatal. I'd check to see if the stderr.log file for the region server process and see if there were any other messages after that Permission Denied. My guess is that the region server log file may contain errors regarding znodes. If Hbase created an Zookeeper znodes while Kerberos was enabled, then the znodes will require auth and will need to be recreated.

I think the solution I mentioned here should help:

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpress-5-5-1-HBase-issue/m-p/42535/highlight/true#M7642

Re: After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

bgooley — Thu, 18 Jul 2019 05:23:29 GMT

@Debashish, let's make sure that we can confirm the cause of your problem first, though, before pursuing the znode stuff.

Re: After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

Debashish — Thu, 18 Jul 2019 05:37:53 GMT

@bgooley

region server error log:

Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ acquire_kerberos_tgt hbase.keytab
+ '[' -z hbase.keytab ']'
+ KERBEROS_PRINCIPAL=
+ '[' '!' -z '' ']'
+ '[' -n '' ']'
+ export 'HBASE_OPTS=-Djava.net.preferIPv4Stack=true '
+ HBASE_OPTS='-Djava.net.preferIPv4Stack=true '
+ locate_hbase_script
+ '[' 6 -ge 5 ']'
+ export BIGTOP_DEFAULTS_DIR=
+ BIGTOP_DEFAULTS_DIR=
+ HBASE_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase
+ '[' upgrade = regionserver ']'
+ '[' region_mover = regionserver ']'
+ '[' toggle_balancer = regionserver ']'
+ '[' shell = regionserver ']'
+ '[' remoteSnapshotTool = regionserver ']'
+ '[' '' '!=' '' ']'
+ '[' regionserver = regionserver -a -n '' ']'
+ '[' start = start -a 6 -gt 4 ']'
+ '[' regionserver = regionserver -o master = regionserver ']'
+ export HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330
+ HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330
+ exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER regionserver start
+ znode_cleanup regionserver
++ date
+ echo 'Wed Jul 17 15:27:01 CAT 2019 Starting znode cleanup thread with HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330 for regionserver'
++ replace_pid -Djava.net.preferIPv4Stack=true
++ echo -Djava.net.preferIPv4Stack=true
++ sed 's#{{PID}}#55330#g'
+ HBASE_OPTS=-Djava.net.preferIPv4Stack=true
+ '[' '' '!=' '' ']'
+ LOG_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/logs/znode_cleanup.log
+ set +x

Re: After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

Debashish — Thu, 18 Jul 2019 05:48:20 GMT

@bgooley thanks for the prompt response.

Region server error log:

Can't open /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ acquire_kerberos_tgt hbase.keytab
+ '[' -z hbase.keytab ']'
+ KERBEROS_PRINCIPAL=
+ '[' '!' -z '' ']'
+ '[' -n '' ']'
+ export 'HBASE_OPTS=-Djava.net.preferIPv4Stack=true '
+ HBASE_OPTS='-Djava.net.preferIPv4Stack=true '
+ locate_hbase_script
+ '[' 6 -ge 5 ']'
+ export BIGTOP_DEFAULTS_DIR=
+ BIGTOP_DEFAULTS_DIR=
+ HBASE_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase
+ '[' upgrade = regionserver ']'
+ '[' region_mover = regionserver ']'
+ '[' toggle_balancer = regionserver ']'
+ '[' shell = regionserver ']'
+ '[' remoteSnapshotTool = regionserver ']'
+ '[' '' '!=' '' ']'
+ '[' regionserver = regionserver -a -n '' ']'
+ '[' start = start -a 6 -gt 4 ']'
+ '[' regionserver = regionserver -o master = regionserver ']'
+ export HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619
+ HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619
+ exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER regionserver start
+ znode_cleanup regionserver
++ date
+ echo 'Wed Jul 17 15:26:58 CAT 2019 Starting znode cleanup thread with HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619 for regionserver'
++ replace_pid -Djava.net.preferIPv4Stack=true
++ echo -Djava.net.preferIPv4Stack=true
++ sed 's#{{PID}}#31619#g'
+ HBASE_OPTS=-Djava.net.preferIPv4Stack=true
+ '[' '' '!=' '' ']'
+ LOG_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/logs/znode_cleanup.log
+ set +x

Re: After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

bgooley — Fri, 19 Jul 2019 01:31:31 GMT

@Debashish, yup, stderr.log looks good. Please check your Region Server log file on that host (usually it is in /var/log/hbase with the name REGIONSERVER in it). I am guessing you will see an attempt to start up and then a failure of sorts.