Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

HBase Master Failed to start ??

Labels:
avatar
author-rank subhash_parise3
Super Collaborator

Created ‎01-29-2016 01:24 PM

Please see the log below:-

2016-01-29 13:00:41,580 FATAL [eu-lamp-dev-xl-0019-hadoop-sec-master:16000.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.ipc.RemoteException(org.apache.ranger.authorization.hadoop.exceptions.RangerAccessControlException): Permission denied: principal{user=hbase,groups: [hadoop]}, access=null, /apps/hbase/data
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:327)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1698)
        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3856)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
        at org.apache.hadoop.ipc.Client.call(Client.java:1427)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy20.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
        at com.sun.proxy.$Proxy21.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:424)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
        at java.lang.Thread.run(Thread.java:745)
2016-01-29 13:00:41,583 FATAL [eu-lamp-dev-xl-0019-hadoop-sec-master:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.ipc.RemoteException(org.apache.ranger.authorization.hadoop.exceptions.RangerAccessControlException): Permission denied: principal{user=hbase,groups: [hadoop]}, access=null, /apps/hbase/data
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:327)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1698)
        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3856)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
        at org.apache.hadoop.ipc.Client.call(Client.java:1427)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy20.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
        at com.sun.proxy.$Proxy21.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:424)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
        at java.lang.Thread.run(Thread.java:745)
2016-01-29 13:00:41,584 INFO  [eu-lamp-dev-xl-0019-hadoop-sec-master:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
2016-01-29 13:00:41,585 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] regionserver.HRegionServer: Stopping infoServer
2016-01-29 13:00:41,610 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010
2016-01-29 13:00:41,612 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] regionserver.HRegionServer: stopping server eu-lamp-dev-xl-0019-hadoop-sec-master,16000,1454072438556
2016-01-29 13:00:41,612 DEBUG [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] zookeeper.MetaTableLocator: Stopping MetaTableLocator
2016-01-29 13:00:41,613 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24fb78762b2015b
2016-01-29 13:00:41,621 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] zookeeper.ZooKeeper: Session: 0x24fb78762b2015b closed
2016-01-29 13:00:41,621 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-01-29 13:00:41,621 DEBUG [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] ipc.AbstractRpcClient: Stopping rpc client
2016-01-29 13:00:41,631 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] regionserver.HRegionServer: stopping server eu-lamp-dev-xl-0019-hadoop-sec-master,16000,1454072438556; all regions closed.
2016-01-29 13:00:41,632 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] hbase.ChoreService: Chore service for: eu-lamp-dev-xl-0019-hadoop-sec-master,16000,1454072438556 had [] on shutdown
2016-01-29 13:00:41,632 DEBUG [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] master.HMaster: Stopping service threads
2016-01-29 13:00:41,646 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] ipc.RpcServer: Stopping server on 16000
2016-01-29 13:00:41,646 INFO  [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping
2016-01-29 13:00:41,649 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2016-01-29 13:00:41,649 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2016-01-29 13:00:41,680 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/rs/eu-lamp-dev-xl-0019-hadoop-sec-master,16000,1454072438556 already deleted, retry=false
2016-01-29 13:00:41,694 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] zookeeper.ZooKeeper: Session: 0x24fb78762b2015a closed
2016-01-29 13:00:41,694 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-01-29 13:00:41,694 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] regionserver.HRegionServer: stopping server eu-lamp-dev-xl-0019-hadoop-sec-master,16000,1454072438556; zookeeper connection closed.
2016-01-29 13:00:41,694 INFO  [master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000] regionserver.HRegionServer: master/eu-lamp-dev-xl-0019-hadoop-sec-master/10.8.7.62:16000 exiting
[hdfs@eu-lamp-dev-xl-0019-hadoop-sec-master hbase]$ hdfs dfs -ls /apps/hbase
Found 2 items
drwxr-xr-x   - hbase hdfs          0 2016-01-11 12:18 /apps/hbase/data
drwx--x--x   - hbase hdfs          0 2015-07-22 16:26 /apps/hbase/staging
[hdfs@eu-lamp-dev-xl-0019-hadoop-sec-master hbase]$ hdfs dfs -ls /apps/hbase/data
Found 10 items
drwxr-xr-x   - hbase hdfs          0 2015-09-10 10:48 /apps/hbase/data/.hbase-snapshot
drwxr-xr-x   - hbase hdfs          0 2016-01-11 12:18 /apps/hbase/data/.tmp
drwxr-xr-x   - hbase hdfs          0 2016-01-11 12:18 /apps/hbase/data/MasterProcWALs
drwxr-xr-x   - hbase hdfs          0 2016-01-12 11:31 /apps/hbase/data/WALs
drwxr-xr-x   - hbase hdfs          0 2015-09-10 10:48 /apps/hbase/data/archive
drwxr-xr-x   - hbase hdfs          0 2015-07-27 14:47 /apps/hbase/data/corrupt
drwxr-xr-x   - hbase hdfs          0 2015-07-22 16:32 /apps/hbase/data/data
-rwxr-xr-x   3 hbase hdfs         42 2015-07-22 16:32 /apps/hbase/data/hbase.id
-rwxr-xr-x   3 hbase hdfs          7 2015-07-22 16:32 /apps/hbase/data/hbase.version
drwxr-xr-x   - hbase hdfs          0 2016-01-28 13:32 /apps/hbase/data/oldWALs
[hdfs@eu-lamp-dev-xl-0019-hadoop-sec-master hbase]$ hdfs dfs -ls /
Found 10 items
drwxrwxrwx   - yarn   hadoop          0 2016-01-04 16:35 /app-logs
drwxrwxrwx   - hdfs   hdfs            0 2015-07-22 16:28 /apps
drwxr-xr-x   - hdfs   hdfs            0 2015-07-22 16:26 /hdp
drwxr-xr-x   - mapred hdfs            0 2015-07-22 16:23 /mapred
drwxrwxrwx   - hdfs   hdfs            0 2015-07-22 16:23 /mr-history
drwxr-xr-x   - hdfs   hdfs            0 2016-01-26 09:23 /sources
drwxr-xr-x   - hdfs   hdfs            0 2015-07-22 16:22 /system
drwxr-xr-x   - hdfs   hdfs            0 2015-10-27 11:22 /test
drwxrwxrwx   - hdfs   hdfs            0 2016-01-14 10:19 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2016-01-07 10:21 /user
[hdfs@eu-lamp-dev-xl-0019-hadoop-sec-master hbase]$ hdfs dfs -ls /apps
Found 2 items
drwxrwxrwx   - hdfs hdfs          0 2015-07-22 16:26 /apps/hbase
drwxr-xr-x   - hdfs hdfs          0 2015-07-22 16:28 /apps/hive
[hdfs@eu-lamp-dev-xl-0019-hadoop-sec-master hbase]$ hdfs dfs -ls /apps/hbase
Found 2 items
drwxr-xr-x   - hbase hdfs          0 2016-01-11 12:18 /apps/hbase/data
drwx--x--x   - hbase hdfs          0 2015-07-22 16:26 /apps/hbase/staging
[hdfs@eu-lamp-dev-xl-0019-hadoop-sec-master hbase]$
4,389 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank subhash_parise3
Super Collaborator

Created ‎01-29-2016 03:26 PM

I have added the below property in hdfs-site.xml to check ACL's in hdfs filesystem and just restart the hdfs , yarn services in ambari resolves the issue.

<property>

<name>dfs.namenode.acls.enabled</name> <value>true</value>

</property>

View solution in original post

2,744 Views
10 REPLIES 10

avatar
author-rank nsabharwal
Master Mentor

Created ‎01-31-2016 02:10 PM

@subhash parise

Take a look on this demo

If you are using Ranger then you don't have to setup anything except user authorization.

1,016 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
View More Announcements