After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

[#1]

Performed the following steps to disable Kerberos

• Zookeeper -> enableSecurity (Enable Kerberos Authentication)-> false
• HDFS -> hadoop.security.authentication -> Simple
• HDFS -> hadoop.security.authorization -> false
• HDFS -> dfs.datanode.address -> from 1004 (for Kerberos) to default
• HDFS -> dfs.datanode.http.address  -> from 1006 (for Kerberos) to default
• HDFS -> Data Directory Permissions -> from 700 to 755
• HBASE -> hbase.security.authentication -> Simple
• HBASE -> hbase.security.authorization -> false

After doing so, all services started except Hbase.

Found this in error logs

Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied.

How to solve this problem?

[#2 · Accepted solution]

@Debashish,

The "supervisor_status" file is not used by hbase, so this message should be non-fatal.  I'd check to see if the stderr.log file for the region server process and see if there were any other messages after that Permission Denied.  My guess is that the region server log file may contain errors regarding znodes.  If Hbase created an Zookeeper znodes while Kerberos was enabled, then the znodes will require auth and will need to be recreated.

I think the solution I mentioned here should help:

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpre...

[#3]

@Debashish, let's make sure that we can confirm the cause of your problem first, though, before pursuing the znode stuff.

[#4]

@bgooley 

region server error log:

Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ acquire_kerberos_tgt hbase.keytab
+ '[' -z hbase.keytab ']'
+ KERBEROS_PRINCIPAL=
+ '[' '!' -z '' ']'
+ '[' -n '' ']'
+ export 'HBASE_OPTS=-Djava.net.preferIPv4Stack=true '
+ HBASE_OPTS='-Djava.net.preferIPv4Stack=true '
+ locate_hbase_script
+ '[' 6 -ge 5 ']'
+ export BIGTOP_DEFAULTS_DIR=
+ BIGTOP_DEFAULTS_DIR=
+ HBASE_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase
+ '[' upgrade = regionserver ']'
+ '[' region_mover = regionserver ']'
+ '[' toggle_balancer = regionserver ']'
+ '[' shell = regionserver ']'
+ '[' remoteSnapshotTool = regionserver ']'
+ '[' '' '!=' '' ']'
+ '[' regionserver = regionserver -a -n '' ']'
+ '[' start = start -a 6 -gt 4 ']'
+ '[' regionserver = regionserver -o master = regionserver ']'
+ export HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330
+ HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330
+ exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER regionserver start
+ znode_cleanup regionserver
++ date
+ echo 'Wed Jul 17 15:27:01 CAT 2019 Starting znode cleanup thread with HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330 for regionserver'
++ replace_pid -Djava.net.preferIPv4Stack=true
++ echo -Djava.net.preferIPv4Stack=true
++ sed 's#{{PID}}#55330#g'
+ HBASE_OPTS=-Djava.net.preferIPv4Stack=true
+ '[' '' '!=' '' ']'
+ LOG_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/logs/znode_cleanup.log
+ set +x

[#5]

@bgooley thanks for the prompt response.

Region server error log: 

Can't open /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ acquire_kerberos_tgt hbase.keytab
+ '[' -z hbase.keytab ']'
+ KERBEROS_PRINCIPAL=
+ '[' '!' -z '' ']'
+ '[' -n '' ']'
+ export 'HBASE_OPTS=-Djava.net.preferIPv4Stack=true '
+ HBASE_OPTS='-Djava.net.preferIPv4Stack=true '
+ locate_hbase_script
+ '[' 6 -ge 5 ']'
+ export BIGTOP_DEFAULTS_DIR=
+ BIGTOP_DEFAULTS_DIR=
+ HBASE_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase
+ '[' upgrade = regionserver ']'
+ '[' region_mover = regionserver ']'
+ '[' toggle_balancer = regionserver ']'
+ '[' shell = regionserver ']'
+ '[' remoteSnapshotTool = regionserver ']'
+ '[' '' '!=' '' ']'
+ '[' regionserver = regionserver -a -n '' ']'
+ '[' start = start -a 6 -gt 4 ']'
+ '[' regionserver = regionserver -o master = regionserver ']'
+ export HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619
+ HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619
+ exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER regionserver start
+ znode_cleanup regionserver
++ date
+ echo 'Wed Jul 17 15:26:58 CAT 2019 Starting znode cleanup thread with HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619 for regionserver'
++ replace_pid -Djava.net.preferIPv4Stack=true
++ echo -Djava.net.preferIPv4Stack=true
++ sed 's#{{PID}}#31619#g'
+ HBASE_OPTS=-Djava.net.preferIPv4Stack=true
+ '[' '' '!=' '' ']'
+ LOG_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/logs/znode_cleanup.log
+ set +x

[#6]

@Debashish, yup, stderr.log looks good.  Please check your Region Server log file on that host (usually it is in /var/log/hbase with the name REGIONSERVER in it).  I am guessing you will see an attempt to start up and then a failure of sorts.