Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

avatar
New Contributor
Performed the following steps to disable Kerberos
  • Zookeeper -> enableSecurity (Enable Kerberos Authentication)-> false
  • HDFS -> hadoop.security.authentication -> Simple
  • HDFS -> hadoop.security.authorization -> false
  • HDFS -> dfs.datanode.address -> from 1004 (for Kerberos) to default
  • HDFS -> dfs.datanode.http.address  -> from 1006 (for Kerberos) to default
  • HDFS -> Data Directory Permissions -> from 700 to 755
  • HBASE -> hbase.security.authentication -> Simple
  • HBASE -> hbase.security.authorization -> false

After doing so, all services started except Hbase.

Found this in error logs

Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied.


How to solve this problem?

1 ACCEPTED SOLUTION

avatar
Master Guru

@Debashish,

 

The "supervisor_status" file is not used by hbase, so this message should be non-fatal.  I'd check to see if the stderr.log file for the region server process and see if there were any other messages after that Permission Denied.  My guess is that the region server log file may contain errors regarding znodes.  If Hbase created an Zookeeper znodes while Kerberos was enabled, then the znodes will require auth and will need to be recreated.

 

I think the solution I mentioned here should help:

 

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpre...

View solution in original post

5 REPLIES 5

avatar
Master Guru

@Debashish,

 

The "supervisor_status" file is not used by hbase, so this message should be non-fatal.  I'd check to see if the stderr.log file for the region server process and see if there were any other messages after that Permission Denied.  My guess is that the region server log file may contain errors regarding znodes.  If Hbase created an Zookeeper znodes while Kerberos was enabled, then the znodes will require auth and will need to be recreated.

 

I think the solution I mentioned here should help:

 

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpre...

avatar
Master Guru

@Debashish, let's make sure that we can confirm the cause of your problem first, though, before pursuing the znode stuff.

avatar
New Contributor

@bgooley 

region server error log:

Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ acquire_kerberos_tgt hbase.keytab
+ '[' -z hbase.keytab ']'
+ KERBEROS_PRINCIPAL=
+ '[' '!' -z '' ']'
+ '[' -n '' ']'
+ export 'HBASE_OPTS=-Djava.net.preferIPv4Stack=true '
+ HBASE_OPTS='-Djava.net.preferIPv4Stack=true '
+ locate_hbase_script
+ '[' 6 -ge 5 ']'
+ export BIGTOP_DEFAULTS_DIR=
+ BIGTOP_DEFAULTS_DIR=
+ HBASE_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase
+ '[' upgrade = regionserver ']'
+ '[' region_mover = regionserver ']'
+ '[' toggle_balancer = regionserver ']'
+ '[' shell = regionserver ']'
+ '[' remoteSnapshotTool = regionserver ']'
+ '[' '' '!=' '' ']'
+ '[' regionserver = regionserver -a -n '' ']'
+ '[' start = start -a 6 -gt 4 ']'
+ '[' regionserver = regionserver -o master = regionserver ']'
+ export HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330
+ HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330
+ exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER regionserver start
+ znode_cleanup regionserver
++ date
+ echo 'Wed Jul 17 15:27:01 CAT 2019 Starting znode cleanup thread with HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330 for regionserver'
++ replace_pid -Djava.net.preferIPv4Stack=true
++ echo -Djava.net.preferIPv4Stack=true
++ sed 's#{{PID}}#55330#g'
+ HBASE_OPTS=-Djava.net.preferIPv4Stack=true
+ '[' '' '!=' '' ']'
+ LOG_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/logs/znode_cleanup.log
+ set +x

 

avatar
New Contributor

@bgooley thanks for the prompt response.

Region server error log: 

Can't open /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ acquire_kerberos_tgt hbase.keytab
+ '[' -z hbase.keytab ']'
+ KERBEROS_PRINCIPAL=
+ '[' '!' -z '' ']'
+ '[' -n '' ']'
+ export 'HBASE_OPTS=-Djava.net.preferIPv4Stack=true '
+ HBASE_OPTS='-Djava.net.preferIPv4Stack=true '
+ locate_hbase_script
+ '[' 6 -ge 5 ']'
+ export BIGTOP_DEFAULTS_DIR=
+ BIGTOP_DEFAULTS_DIR=
+ HBASE_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase
+ '[' upgrade = regionserver ']'
+ '[' region_mover = regionserver ']'
+ '[' toggle_balancer = regionserver ']'
+ '[' shell = regionserver ']'
+ '[' remoteSnapshotTool = regionserver ']'
+ '[' '' '!=' '' ']'
+ '[' regionserver = regionserver -a -n '' ']'
+ '[' start = start -a 6 -gt 4 ']'
+ '[' regionserver = regionserver -o master = regionserver ']'
+ export HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619
+ HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619
+ exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER regionserver start
+ znode_cleanup regionserver
++ date
+ echo 'Wed Jul 17 15:26:58 CAT 2019 Starting znode cleanup thread with HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/znode31619 for regionserver'
++ replace_pid -Djava.net.preferIPv4Stack=true
++ echo -Djava.net.preferIPv4Stack=true
++ sed 's#{{PID}}#31619#g'
+ HBASE_OPTS=-Djava.net.preferIPv4Stack=true
+ '[' '' '!=' '' ']'
+ LOG_FILE=/var/run/cloudera-scm-agent/process/762-hbase-REGIONSERVER/logs/znode_cleanup.log
+ set +x

 

avatar
Master Guru

@Debashish, yup, stderr.log looks good.  Please check your Region Server log file on that host (usually it is in /var/log/hbase with the name REGIONSERVER in it).  I am guessing you will see an attempt to start up and then a failure of sorts.