After upgrading to 3.0.1 and enabling kerberos, Hbase master is failing to start due to missing 'master' znode. It is leading to Atlas start failure.
atlas TABLE Took 8.2656 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master 2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus ATLAS_ENTITY_AUDIT_EVENTS atlas TABLE Took 8.2337 secondsjava exception
In zookeeper 'master' znode is present is hbase-unsecure but not in secure-hbase
[zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure [replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table] [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table]
How do I fix this?
Is this a known issue in 3.0.1?
Thanks in advance.
Solution for the desparate souls like me.
copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop
Restart Hbase and Atlas.
Don't know if its the right thing but worked!!! Start to laugh again !!!.