Member since
12-21-2015
12
Posts
7
Kudos Received
0
Solutions
02-22-2017
08:26 PM
Thanks Josh for explaining! A follow question would be what does Master "fail" mean? Sometimes we notice our app talking HBase does not respond. And, then we notice that the Hbase master it was talking to had a resource alert on Ambari UI. And, we do not know if the failover was even initiated(only when one master crashes? What about if it is just slow?). When does failover get initiated? Can you point to some article which talks about it? I can't seem to find one easily online. We also tried to do this: 1. talk to the active master ; Application writehbase-hbase-master-regionserver02stormca-internall.zips to the active master some data. Confirmed that the data was there in the active master by scanning the table on hbase shell. 2. Then failed over (to simulate failover) artificially. Then we did the same as above. List shows the table. But, scan on the table fails. (no response). Attached the log with relevant errors.
... View more
02-22-2017
12:19 AM
We are running a Hbase Cluster with HBase slave and master configuration. We need a way to make sure when Hbase master active fails, the request from application should automatically be redirected to Hbase master in failover more. We run our cluster in ec2. We need some typical strategies used for this purpose. Any ideas ?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
09-27-2016
08:37 PM
@Enis Thanks for the input. I am running into the same problem. But, I copied and added the hbase-site.xml to my mac host laptop. Are other file needed as well within (under like: /etc/hbase/conf/core-site.xml)And, now I am running into : LOG of relevant properties set correctly: conf.Configuration (Configuration.java:getConfResourceAsInputStream(2355)) - found resource hbase-site.xml at file:/etc/hbase/conf/hbase-site.xml
2016-09-27 12:21:22,651 INFO [main] hbasetest.ConnectHBase (ConnectHBase.java:main(61)) - hbase.zookeeper.quorum:/hbase-unsecure
2016-09-27 12:21:22,651 INFO [main] hbasetest.ConnectHBase (ConnectHBase.java:main(62)) - hbase.zookeeper.quorum:hdfs://sandbox.hortonworks.com:8020
2016-09-27 12:21:22,651 INFO [main] hbasetest.ConnectHBase (ConnectHBase.java:main(63)) - hbase.zookeeper.quorum:2181 Exception: <code> where applicable
2016-09-27 12:21:28,045 INFO [main] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=hconnection-0x22c86919 connecting to ZooKeeper ensemble=sandbox.hortonworks.com:2181
2016-09-27 12:21:31,461 INFO [main] client.ConnectionManager$HConnectionImplementation (ConnectionManager.java:closeZooKeeperWatcher(1682)) - Closing zookeeper sessionid=0x1569fdbdb97003f
Exception in thread "main" org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1539)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1559)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isMasterRunning(ConnectionManager.java:928)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:2877)
at com.plantronics.data.storm.hbasetest.ConnectHBase.main(ConnectHBase.java:68)
Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:53657)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1570)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1508)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1530)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:410)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:716)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1200)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
... 10 more
Disconnected from the target VM, address: '127.0.0.1:51389', transport: 'socket' </code> Wondering if this is port opening issue. Is there a horton help page similar to this for Seeing a exception for connection opening here: java.nio.channels.SocketChannel[connection-pending remote=sandbox.hortonworks.com/192.168.56.101:16000] sandbox.hortonworks.com 16000 (Not able to connect here) http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0-Win/bk_HDP_Install_Win/content/ref-79239257-778e-42a9-9059-d982d0c08885.1.html. for mac/linux?
... View more
05-26-2016
10:12 PM
Hi, Very useful article. I tried this exact code snippet. But, failing. Any help will be appreciated : I am no HDP 2.4. Exception in thread "main" java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:304)
at com.plantronics.data.storm.hbasetest.ConnectHBase.main(ConnectHBase.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:399)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:552)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1191)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1158)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
... 14 more
... View more
05-04-2016
08:50 PM
3 Kudos
Anyone has seen kafka broker running into GC related CPU high ? We see it being 100%. And, we have no load on kafka. And, we see the following activity going on throughout that time : [GC (Allocation Failure) 4.651: [ParNew: 272640K->7265K(306688K), 0.0277514 secs]272640K->7265K(1014528K), 0.0281243 secs] [Times: user=0.03 sys=0.05, real=0.03 secs] There is a ticket here : https://issues.apache.org/jira/browse/KAFKA-2627 Heap memory setting on broker : java -Xmx1G -Xms1G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC
... View more
Labels:
- Labels:
-
Apache Kafka
04-29-2016
06:10 PM
1 Kudo
Most topics (created by our app) are following the policy(deletes the data after 2 days). But, there is one which is internal to kafka which is used for consumer offset position(We use the client api to commit the offset). This is not being deleted after 2 days. kakfka broker version: 2.3.4.0-3485. Here are the retention policy values:
Converted the server.properties from kafka config to a txt and attaching it here: Kafka broker retention policy configuration(from server.properties)server.txt: ############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion
log.retention.hours=48
# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=536870912
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
# By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires.
# If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction.
log.cleaner.enable=false
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Storm