Skip to main content
Debashish
Debashish
Silver ⭐️⭐️
July 18, 2019
Solved

After disabling Kerberos on Cloudera 6.2.0, Unable to start Hbase

Performed the following steps to disable Kerberos
  • Zookeeper -> enableSecurity (Enable Kerberos Authentication)-> false
  • HDFS -> hadoop.security.authentication -> Simple
  • HDFS -> hadoop.security.authorization -> false
  • HDFS -> dfs.datanode.address -> from 1004 (for Kerberos) to default
  • HDFS -> dfs.datanode.http.address  -> from 1006 (for Kerberos) to default
  • HDFS -> Data Directory Permissions -> from 700 to 755
  • HBASE -> hbase.security.authentication -> Simple
  • HBASE -> hbase.security.authorization -> false

After doing so, all services started except Hbase.

Found this in error logs

Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied.


How to solve this problem?

bgooley
bgooleyAnswer
Gold ⭐️
July 18, 2019

@Debashish,

 

The "supervisor_status" file is not used by hbase, so this message should be non-fatal.  I'd check to see if the stderr.log file for the region server process and see if there were any other messages after that Permission Denied.  My guess is that the region server log file may contain errors regarding znodes.  If Hbase created an Zookeeper znodes while Kerberos was enabled, then the znodes will require auth and will need to be recreated.

 

I think the solution I mentioned here should help:

 

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpress-5-5-1-HBase-issue/m-p/42535/highlight/true#M7642

View in thread

5 replies

bgooley
Gold ⭐️
bgooleyAnswer
Gold ⭐️
July 18, 2019

@Debashish,

 

The "supervisor_status" file is not used by hbase, so this message should be non-fatal.  I'd check to see if the stderr.log file for the region server process and see if there were any other messages after that Permission Denied.  My guess is that the region server log file may contain errors regarding znodes.  If Hbase created an Zookeeper znodes while Kerberos was enabled, then the znodes will require auth and will need to be recreated.

 

I think the solution I mentioned here should help:

 

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpress-5-5-1-HBase-issue/m-p/42535/highlight/true#M7642

Translate
bgooley
Gold ⭐️
bgooley
Gold ⭐️
July 18, 2019

@Debashish, let's make sure that we can confirm the cause of your problem first, though, before pursuing the znode stuff.

Translate
    Debashish
    DebashishAuthor
    Silver ⭐️⭐️
    July 18, 2019

    @bgooley 

    region server error log:

    Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied.
    + make_scripts_executable
    + find /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
    + acquire_kerberos_tgt hbase.keytab
    + '[' -z hbase.keytab ']'
    + KERBEROS_PRINCIPAL=
    + '[' '!' -z '' ']'
    + '[' -n '' ']'
    + export 'HBASE_OPTS=-Djava.net.preferIPv4Stack=true '
    + HBASE_OPTS='-Djava.net.preferIPv4Stack=true '
    + locate_hbase_script
    + '[' 6 -ge 5 ']'
    + export BIGTOP_DEFAULTS_DIR=
    + BIGTOP_DEFAULTS_DIR=
    + HBASE_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase
    + '[' upgrade = regionserver ']'
    + '[' region_mover = regionserver ']'
    + '[' toggle_balancer = regionserver ']'
    + '[' shell = regionserver ']'
    + '[' remoteSnapshotTool = regionserver ']'
    + '[' '' '!=' '' ']'
    + '[' regionserver = regionserver -a -n '' ']'
    + '[' start = start -a 6 -gt 4 ']'
    + '[' regionserver = regionserver -o master = regionserver ']'
    + export HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330
    + HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330
    + exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER regionserver start
    + znode_cleanup regionserver
    ++ date
    + echo 'Wed Jul 17 15:27:01 CAT 2019 Starting znode cleanup thread with HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/znode55330 for regionserver'
    ++ replace_pid -Djava.net.preferIPv4Stack=true
    ++ echo -Djava.net.preferIPv4Stack=true
    ++ sed 's#{{PID}}#55330#g'
    + HBASE_OPTS=-Djava.net.preferIPv4Stack=true
    + '[' '' '!=' '' ']'
    + LOG_FILE=/var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/logs/znode_cleanup.log
    + set +x

     

    Translate
    Terms & ConditionsAccessibility statement