Member since
11-13-2017
50
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5055 | 11-15-2018 12:35 PM |
11-15-2018
12:35 PM
Solution for the desparate souls like me. copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop Restart Hbase and Atlas. Don't know if its the right thing but worked!!! Start to laugh again !!!.
... View more
11-15-2018
12:16 PM
Solution for the desparate souls like me. copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop Restart Hbase and Atlas. Don't know if its the right thing but worked!!! Start to laugh again !!!.
... View more
11-15-2018
10:07 AM
Hi, After upgrading to 3.0.1 and enabling kerberos, Hbase master is failing to start due to missing 'master' znode. It is leading to Atlas start failure. atlas
TABLE
Took 8.2656 secondsjava exception
ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master
2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus
ATLAS_ENTITY_AUDIT_EVENTS
atlas
TABLE
Took 8.2337 secondsjava exception In zookeeper 'master' znode is present is hbase-unsecure but not in secure-hbase [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure
[replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table]
[zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table] How do I fix this? Is this a known issue in 3.0.1? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache HBase
11-14-2018
05:29 PM
Some more logs atlas
TABLE
Took 8.2656 secondsjava exception
ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master
2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus
ATLAS_ENTITY_AUDIT_EVENTS
atlas
TABLE
Took 8.2337 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master
... View more
11-14-2018
04:31 PM
2018-11-14 16:15:03,009 WARN [master/hdata4:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
2018-11-14 16:15:03,009 INFO [master/hdata4:16000] assignment.AssignmentManager: Stopping assignment manager
2018-11-14 16:15:03,032 WARN [master/hdata4:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions.
2018-11-14 16:15:03,033 INFO [master/hdata4:16000] procedure2.RemoteProcedureDispatcher: Stopping procedure remote dispatcher
2018-11-14 16:15:03,033 INFO [master/hdata4:16000] procedure2.ProcedureExecutor: Stopping
2018-11-14 16:15:03,036 INFO [master/hdata4:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store, isAbort=false
2018-11-14 16:15:03,070 ERROR [master/hdata4:16000] wal.WALProcedureStore: Unable to close the stream
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Client (=DFSClient_NONMAPREDUCE_-826562846_1) is not the lease owner (=DFSClient_NONMAPREDUCE_-1195801889_1: /apps/hbase/data/MasterProcWALs/pv2-00000000000000000011.log (inode 10368947) Holder DFSClient_NONMAPREDUCE_-826562846_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2837)
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:685)
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:671)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2858)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:928)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:607)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.complete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:550)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy19.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372)
at com.sun.proxy.$Proxy20.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372)
at com.sun.proxy.$Proxy20.complete(Unknown Source)
2018-11-14 16:15:03,071 INFO [master/hdata4:16000] hbase.ChoreService: Chore service for: master/hdata4:16000.splitLogManager. had [] on shutdown
2018-11-14 16:15:03,071 INFO [master/hdata4:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2018-11-14 16:15:03,071 ERROR [master/hdata4:16000] access.TableAuthManager: Something wrong with the TableAuthManager reference counting: org.apache.hadoop.hbase.security.access.TableAuthManager@7e83992 whose count is null
Zookeeper znode listing for unsecure and secure. master znode is missing from hbase-secure [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure
[replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table]
[zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table]
... View more
11-12-2018
02:20 PM
After enablling kerberos HBase master failing to restart. On investigation found that in zookeeper /hbase-secure/master node is missing. Zookeeper shows following Acl [zk: localhost:2181(CONNECTED) 1] getAcl /hbase-secure 'world,'anyone
: r 'sasl,'hbase
: cdrwa 'sasl,'hbase
: cdrwa Tried all the threads eg. https://community.hortonworks.com/content/supportkb/151088/how-to-force-remove-znode-with-stale-acl.html https://community.hortonworks.com/articles/82405/how-to-remove-acl-protected-zk-node.html Any ideas?
... View more
Labels:
- Labels:
-
Apache HBase
11-08-2018
11:53 AM
Many Thanks Sandeep. Yes it is working now. I would like to add more details for fix: Update hive-interactive-env template and hive-env template Hive config properties from Ambari UI.
... View more
11-07-2018
03:15 PM
Hi Our Hive Server2 Interactive is stopping after around 10 min after restart. I couldn't find hiveserver2Interactive.log in /var/log/hive, hive-server2-interactive.err is showing following: /usr/hdp/current/hive-server2/conf_llap//hive-env.sh: line 43: [: !=: unary operator expected
2018-11-07 14:40:33: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = c4e8edc5-ed5e-4f96-ba70-00779ac20dc9
Hive Session ID = b5ad883d-4a65-45f7-a6dd-1ea3797724ea Any help will be appreciated. Thanks
... View more
Labels:
- Labels:
-
Apache Hive
07-26-2018
11:23 AM
Following jar files should be present gmetric4j-1.0.3.jar jmxetric-1.0.4.jar oncrpc-1.0.7.jar Since I did the version upgrade from 2.5 to 2.6 I copied them from /usr/hdp/2.5.3.0-37/storm/contrib/storm-jmxetric/lib folder Hope this helps
... View more
06-07-2018
09:33 AM
Thanks Gaurav, it was helpful.
... View more