Member since
01-21-2016
290
Posts
76
Kudos Received
3
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3200 | 01-09-2017 11:00 AM | |
| 1262 | 12-15-2016 09:48 AM | |
| 5475 | 02-03-2016 07:00 AM |
05-23-2017
12:56 PM
I am trying to access phoenix table from spark and have enabled pheonix namespace on the server. when i try to access using the phoenix client (sqlline.py), i am able to do without issue. But when i try to access usign spark-shell i get the foll0wing error. java.sql.SQLException: ERROR 726 (43M10): Inconsistent namespace mapping properites.. Cannot initiate connection as SYSTEM:CATALOG is found but client does not have phoenix.schema.isNamespaceMappingEnabled enabled Even i tried printing the classpath of spark-shell and i do see the hbase-site.xml in the classpath. Spark Command: /usr/java/jdk1.8.0_60//bin/java -Dhdp.version=2.6.0.3-8 -cp /usr/hdp/current/phoenix-client/phoenix-4.8.1-HBase-1.1-client.jar:/etc/hbase/conf/hbase-site.xml:/usr/hdp/current/spark-client/conf/:/usr/hdp/current/spark-client/lib/spark-assembly-1.6.3.2.6.0.3-8-hadoop2.7.3.2.6.0.3-8.jar:/usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar:/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar:/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar -Dscala.usejavacp=true -Xms1g -Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name Spark shell What could be the issue. How do i pass the the phoenix namespace enabled parameter from spark-shell?
... View more
Labels:
- Labels:
-
Apache Phoenix
-
Apache Spark
05-03-2017
05:41 AM
Hi, I am using HDP 2.6. Can someone point me to the doc i should refer for the hbase back up and incremental backup.
... View more
Labels:
- Labels:
-
Apache HBase
04-18-2017
11:43 AM
Hi @Josh Elser, this is what i could see in the logs in debug mode. I am seeing a ipc timeout exception. rpc timeout is set around 15 mins. Any other parameter that i should look in? 2017-04-18 10:41:51,628 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e4158de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
2017-04-18 10:41:51,686 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: ClusterId : a6d99271-51ab-41ed-a2d5-93049dedd72d
2017-04-18 10:41:51,693 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is initializing
2017-04-18 10:41:51,760 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/flush-table-proc/acquired already exists
2017-04-18 10:41:51,782 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is initialized
2017-04-18 10:41:51,782 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initializing
2017-04-18 10:41:51,786 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/online-snapshot/acquired already exists
2017-04-18 10:41:51,789 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initialized
2017-04-18 10:41:51,793 WARN [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.lowerLimit is deprecated. Instead use hbase.regionserver.global.memstore.size.lower.limit
2017-04-18 10:41:51,795 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.MemStoreFlusher: globalMemStoreLimit=1.6 G, globalMemStoreLimitLowMark=1003.6 M, maxHeap=3.9 G
2017-04-18 10:41:51,800 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: CompactionChecker runs every 10sec
2017-04-18 10:41:51,818 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ea71264, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=abcd013.abcdsolutions.local/172.16.3.197:0
2017-04-18 10:41:51,823 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020
2017-04-18 10:41:51,870 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled
2017-04-18 10:41:51,870 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled
2017-04-18 10:41:51,877 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: reportForDuty to master=abcd011.abcdsolutions.local,16000,1492492302253 with port=16020, startcode=1492492307927
2017-04-18 10:41:52,101 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Use SIMPLE authentication for service RegionServerStatusService, sasl=false
2017-04-18 10:41:52,154 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Connecting to abcd011.abcdsolutions.local/172.16.3.195:16000
2017-04-18 10:42:02,024 WARN [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.io.IOException: Call to abcd011.abcdsolutions.local/172.16.3.195:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=0, waitTime=10001, operationTimeout=10000 expired.
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:10859)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2406)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:936)
... View more
04-18-2017
11:30 AM
Hi @Josh Elser, what i could see from this error failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException::Call id=0, waitTime=10001, operationTimeout=10000 expired is a timeout exception. The property for rpc timeout is having this value <property>
<name>hbase.rpc.timeout</name>
<value>1800000</value>
</property>
Is there any other property that needs to be set. one more fact is we are having hbase setup in HA mode and after upgrade the service got restarted. so there is a change of hbase active master after restart.
... View more
04-18-2017
11:30 AM
Hi @Josh Elser, I couldnt find anything in the debug logs 2017-04-18 10:41:51,628 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e4158de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
2017-04-18 10:41:51,686 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: ClusterId : a6d99271-51ab-41ed-a2d5-93049dedd72d
2017-04-18 10:41:51,693 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is initializing
2017-04-18 10:41:51,760 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/flush-table-proc/acquired already exists
2017-04-18 10:41:51,782 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is initialized
2017-04-18 10:41:51,782 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initializing
2017-04-18 10:41:51,786 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/online-snapshot/acquired already exists
2017-04-18 10:41:51,789 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initialized
2017-04-18 10:41:51,793 WARN [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.lowerLimit is deprecated. Instead use hbase.regionserver.global.memstore.size.lower.limit
2017-04-18 10:41:51,795 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.MemStoreFlusher: globalMemStoreLimit=1.6 G, globalMemStoreLimitLowMark=1003.6 M, maxHeap=3.9 G
2017-04-18 10:41:51,800 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: CompactionChecker runs every 10sec
2017-04-18 10:41:51,818 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ea71264, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=abcd013.abcdsolutions.local/172.16.3.197:0
2017-04-18 10:41:51,823 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020
2017-04-18 10:41:51,870 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled
2017-04-18 10:41:51,870 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled
2017-04-18 10:41:51,877 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: reportForDuty to master=abcd011.abcdsolutions.local,16000,1492492302253 with port=16020, startcode=1492492307927
2017-04-18 10:41:52,101 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Use SIMPLE authentication for service RegionServerStatusService, sasl=false
2017-04-18 10:41:52,154 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Connecting to abcd011.abcdsolutions.local/172.16.3.195:16000
2017-04-18 10:42:02,024 WARN [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.io.IOException: Call to abcd011.abcdsolutions.local/172.16.3.195:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=0, waitTime=10001, operationTimeout=10000 expired.
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:10859)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2406)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:936)
... View more
04-17-2017
03:37 AM
Hi @Josh Elser, this is what i see in the Hbase master logs master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 312811771 ms, expecting minimum of 1, maximum of 2147483647, timeout of 30000 ms, interval of 1500 ms.
... View more
04-17-2017
03:32 AM
Hi @Josh Elser, i am able to telnet to the master without any issues. Also the master is running fine.
... View more
04-13-2017
12:59 PM
We upgraded from HDP 2.4 to 2.6. All services are starting. but hbase region servers are not able to communicate to the master with the following exception. any htoughts. nothing has changed after upgrade /etc/hosts entries are intact 2017-04-13 18:13:38,481 WARN [regionserver/172.16.3.197:16020] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.io.IOException: Call to abcdsolutions.local/172.16.3.196:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=9, waitTime=10001, operationTimeout=10000 expired.
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:10859)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2406)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:936)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Call to abcdsolutions.local/172.16.3.196:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=9, waitTime=10001, operationTimeout=10000 expired.
at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
... 5 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=9, waitTime=10001, operationTimeout=10000 expired.
at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
... 6 more
... View more
Labels:
- Labels:
-
Apache HBase
04-05-2017
06:51 AM
Thanks @Jay SenSharma, It helps, our case is more of realtime in nature. so the usage of sqoop is not possible
... View more
04-05-2017
06:19 AM
I have a use case where i need to move data from a mysql database to Hbase on a realtime basis using kafka as the intemediate data store. Will Kafka connect help to achieve this usecase.
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Kafka