Reply
Contributor
Posts: 31
Registered: ‎07-11-2017

Hbase connection : Windows to region servers with edge node in between (using java api)

Hi,

 

I am trying to connect to Hbase table using JAVA application that is running on windows machine.I could establish connection to the table using zookeeper thats running on the edge node and get the region server IP address from the .meta table. Then the client application is trying to connect directly to the region server (data node)  from windows machine and throwing the below exceptions.

 

Due to edge node we cannot connect from windows to data node directly and also edge node and other nodes (master and data nodes) are in different network , how can this issue be resolved?

 

17/08/04 09:18:34 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67)
at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81)
at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96)
at com.seagate.dars.imagedefectanalysis.dao.HBaseDao.saveFiles2FS(HBaseDao.java:138)
at com.seagate.dars.imagedefectanalysis.ImageDefectAnalysis.main(ImageDefectAnalysis.java:40)
17/08/04 09:18:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/04 09:18:34 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0xba8d91c connecting to ZooKeeper ensemble=en01.com:2181
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:host.name=NRM-UabcL003.ad.seagate.com
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_112
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:java.home=C:\Users\abc\AppData\Local\MyEclipse 2017 CI\binary\com.sun.java.jdk8.win32.x86_64_1.8.0.v112\jre
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:java.class.path=C:\Users\abc\renuka\Workspaces\MyEclipse 2017 CI\ImageDefectAnalysis_original\target\classes;C:\Users\abc\.m2\repository\org\apache\hbase\hbase-client\1.0.0\hbase-client-1.0.0.jar;C:\Users\abc\.m2\repository\org\apache\hbase\hbase-annotations\1.0.0\hbase-annotations-1.0.0.jar;C:\Users\abc\AppData\Local\MyEclipse 2017 CI\binary\com.sun.java.jdk8.win32.x86_64_1.8.0.v112\lib\tools.jar;C:\Users\abc\.m2\repository\org\apache\hbase\hbase-common\1.0.0\hbase-common-1.0.0.jar;C:\Users\abc\.m2\repository\org\apache\hbase\hbase-protocol\1.0.0\hbase-protocol-1.0.0.jar;C:\Users\abc\.m2\repository\commons-codec\commons-codec\1.9\commons-codec-1.9.jar;C:\Users\abc\.m2\repository\commons-io\commons-io\2.4\commons-io-2.4.jar;C:\Users\abc\.m2\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;C:\Users\abc\.m2\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;C:\Users\abc\.m2\repository\com\google\guava\guava\12.0.1\guava-12.0.1.jar;C:\Users\abc\.m2\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;C:\Users\abc\.m2\repository\io\netty\netty-all\4.0.23.Final\netty-all-4.0.23.Final.jar;C:\Users\abc\.m2\repository\org\apache\zookeeper\zookeeper\3.4.6\zookeeper-3.4.6.jar;C:\Users\abc\.m2\repository\org\apache\htrace\htrace-core\3.1.0-incubating\htrace-core-3.1.0-incubating.jar;C:\Users\abc\.m2\repository\org\codehaus\jackson\jackson-mapper-asl\1.8.8\jackson-mapper-asl-1.8.8.jar;C:\Users\abc\.m2\repository\org\jruby\jcodings\jcodings\1.0.8\jcodings-1.0.8.jar;C:\Users\abc\.m2\repository\org\jruby\joni\joni\2.1.2\joni-2.1.2.jar;C:\Users\abc\.m2\repository\org\apache\hadoop\hadoop-auth\2.5.1\hadoop-auth-2.5.1.jar;C:\Users\abc\.m2\repository\org\apache\httpcomponents\httpclient\4.2.5\httpclient-4.2.5.jar;C:\Users\abc\.m2\repository\org\apache\directory\server\apacheds-kerberos-codec\2.0.0-M15\apacheds-kerberos-codec-2.0.0-M15.jar;C:\Users\abc\.m2\repository\org\apache\directory\server\apacheds-i18n\2.0.0-M15\apacheds-i18n-2.0.0-M15.jar;C:\Users\abc\.m2\repository\org\apache\directory\api\api-asn1-api\1.0.0-M20\api-asn1-api-1.0.0-M20.jar;C:\Users\abc\.m2\repository\org\apache\directory\api\api-util\1.0.0-M20\api-util-1.0.0-M20.jar;C:\Users\abc\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-core\2.5.1\hadoop-mapreduce-client-core-2.5.1.jar;C:\Users\abc\.m2\repository\org\apache\hadoop\hadoop-yarn-common\2.5.1\hadoop-yarn-common-2.5.1.jar;C:\Users\abc\.m2\repository\org\apache\hadoop\hadoop-yarn-api\2.5.1\hadoop-yarn-api-2.5.1.jar;C:\Users\abc\.m2\repository\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar;C:\Users\abc\.m2\repository\javax\xml\stream\stax-api\1.0-2\stax-api-1.0-2.jar;C:\Users\abc\.m2\repository\io\netty\netty\3.6.2.Final\netty-3.6.2.Final.jar;C:\Users\abc\.m2\repository\com\github\stephenc\findbugs\findbugs-annotations\1.3.9-1\findbugs-annotations-1.3.9-1.jar;C:\Users\abc\.m2\repository\junit\junit\4.11\junit-4.11.jar;C:\Users\abc\.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;C:\Users\abc\.m2\repository\org\apache\hadoop\hadoop-hdfs\2.7.1\hadoop-hdfs-2.7.1.jar;C:\Users\abc\.m2\repository\org\mortbay\jetty\jetty\6.1.26\jetty-6.1.26.jar;C:\Users\abc\.m2\repository\org\mortbay\jetty\jetty-util\6.1.26\jetty-util-6.1.26.jar;C:\Users\abc\.m2\repository\com\sun\jersey\jersey-core\1.9\jersey-core-1.9.jar;C:\Users\abc\.m2\repository\com\sun\jersey\jersey-server\1.9\jersey-server-1.9.jar;C:\Users\abc\.m2\repository\asm\asm\3.1\asm-3.1.jar;C:\Users\abc\.m2\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;C:\Users\abc\.m2\repository\commons-daemon\commons-daemon\1.0.13\commons-daemon-1.0.13.jar;C:\Users\abc\.m2\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;C:\Users\abc\.m2\repository\javax\servlet\servlet-api\2.5\servlet-api-2.5.jar;C:\Users\abc\.m2\repository\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;C:\Users\abc\.m2\repository\xmlenc\xmlenc\0.52\xmlenc-0.52.jar;C:\Users\abc\.m2\repository\xerces\xercesImpl\2.9.1\xercesImpl-2.9.1.jar;C:\Users\abc\.m2\repository\xml-apis\xml-apis\1.3.04\xml-apis-1.3.04.jar;C:\Users\abc\.m2\repository\org\fusesource\leveldbjni\leveldbjni-all\1.8\leveldbjni-all-1.8.jar;C:\Users\abc\.m2\repository\org\apache\hadoop\hadoop-common\2.7.1\hadoop-common-2.7.1.jar;C:\Users\abc\.m2\repository\org\apache\hadoop\hadoop-annotations\2.7.1\hadoop-annotations-2.7.1.jar;C:\Users\abc\.m2\repository\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;C:\Users\abc\.m2\repository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;C:\Users\abc\.m2\repository\commons-net\commons-net\3.1\commons-net-3.1.jar;C:\Users\abc\.m2\repository\commons-collections\commons-collections\3.2.1\commons-collections-3.2.1.jar;C:\Users\abc\.m2\repository\javax\servlet\jsp\jsp-api\2.1\jsp-api-2.1.jar;C:\Users\abc\.m2\repository\com\sun\jersey\jersey-json\1.9\jersey-json-1.9.jar;C:\Users\abc\.m2\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;C:\Users\abc\.m2\repository\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar;C:\Users\abc\.m2\repository\org\codehaus\jackson\jackson-jaxrs\1.8.3\jackson-jaxrs-1.8.3.jar;C:\Users\abc\.m2\repository\org\codehaus\jackson\jackson-xc\1.8.3\jackson-xc-1.8.3.jar;C:\Users\abc\.m2\repository\net\java\dev\jets3t\jets3t\0.9.0\jets3t-0.9.0.jar;C:\Users\abc\.m2\repository\org\apache\httpcomponents\httpcore\4.1.2\httpcore-4.1.2.jar;C:\Users\abc\.m2\repository\com\jamesmurty\utils\java-xmlbuilder\0.4\java-xmlbuilder-0.4.jar;C:\Users\abc\.m2\repository\commons-configuration\commons-configuration\1.6\commons-configuration-1.6.jar;C:\Users\abc\.m2\repository\commons-digester\commons-digester\1.8\commons-digester-1.8.jar;C:\Users\abc\.m2\repository\commons-beanutils\commons-beanutils\1.7.0\commons-beanutils-1.7.0.jar;C:\Users\abc\.m2\repository\commons-beanutils\commons-beanutils-core\1.8.0\commons-beanutils-core-1.8.0.jar;C:\Users\abc\.m2\repository\org\slf4j\slf4j-api\1.7.10\slf4j-api-1.7.10.jar;C:\Users\abc\.m2\repository\org\slf4j\slf4j-log4j12\1.7.10\slf4j-log4j12-1.7.10.jar;C:\Users\abc\.m2\repository\org\apache\avro\avro\1.7.4\avro-1.7.4.jar;C:\Users\abc\.m2\repository\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;C:\Users\abc\.m2\repository\org\xerial\snappy\snappy-java\1.0.4.1\snappy-java-1.0.4.1.jar;C:\Users\abc\.m2\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;C:\Users\abc\.m2\repository\com\jcraft\jsch\0.1.42\jsch-0.1.42.jar;C:\Users\abc\.m2\repository\org\apache\curator\curator-client\2.7.1\curator-client-2.7.1.jar;C:\Users\abc\.m2\repository\org\apache\curator\curator-recipes\2.7.1\curator-recipes-2.7.1.jar;C:\Users\abc\.m2\repository\org\apache\curator\curator-framework\2.7.1\curator-framework-2.7.1.jar;C:\Users\abc\.m2\repository\com\google\code\findbugs\jsr305\3.0.0\jsr305-3.0.0.jar;C:\Users\abc\.m2\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;C:\Users\abc\.m2\repository\org\tukaani\xz\1.0\xz-1.0.jar;C:\Users\abc\.m2\repository\org\apache\hadoop\hadoop-core\1.2.1\hadoop-core-1.2.1.jar;C:\Users\abc\.m2\repository\org\apache\commons\commons-math\2.1\commons-math-2.1.jar;C:\Users\abc\.m2\repository\tomcat\jasper-runtime\5.5.12\jasper-runtime-5.5.12.jar;C:\Users\abc\.m2\repository\tomcat\jasper-compiler\5.5.12\jasper-compiler-5.5.12.jar;C:\Users\abc\.m2\repository\org\mortbay\jetty\jsp-api-2.1\6.1.14\jsp-api-2.1-6.1.14.jar;C:\Users\abc\.m2\repository\org\mortbay\jetty\servlet-api-2.5\6.1.14\servlet-api-2.5-6.1.14.jar;C:\Users\abc\.m2\repository\org\mortbay\jetty\jsp-2.1\6.1.14\jsp-2.1-6.1.14.jar;C:\Users\abc\.m2\repository\ant\ant\1.6.5\ant-1.6.5.jar;C:\Users\abc\.m2\repository\commons-el\commons-el\1.0\commons-el-1.0.jar;C:\Users\abc\.m2\repository\hsqldb\hsqldb\1.8.0.10\hsqldb-1.8.0.10.jar;C:\Users\abc\.m2\repository\oro\oro\2.0.8\oro-2.0.8.jar;C:\Users\abc\.m2\repository\org\eclipse\jdt\core\3.1.1\core-3.1.1.jar;C:\Users\abc\.m2\repository\com\sun\mail\javax.mail\1.5.1\javax.mail-1.5.1.jar;C:\Users\abc\.m2\repository\javax\activation\activation\1.1\activation-1.1.jar
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:java.library.path=C:\Users\abc\AppData\Local\MyEclipse 2017 CI\binary\com.sun.java.jdk8.win32.x86_64_1.8.0.v112\bin;C:\windows\Sun\Java\bin;C:\windows\system32;C:\windows;C:\tibco\tibrv\8.2\bin;c:\Program Files (x86)\RSA SecurID Token Common;C:\ProgramData\Oracle\Java\javapath;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files\PuTTY\;C:\Users\abc\AppData\Local\Microsoft\WindowsApps;;.
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=C:\Users\abc\AppData\Local\Temp\1\
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:os.name=Windows 10
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:os.version=10.0
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:user.name=abc
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:user.home=C:\Users\abc
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Client environment:user.dir=C:\Users\abc\Workspaces\MyEclipse 2017 CI\javacode
17/08/04 09:18:34 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=en01.com:2181 sessionTimeout=90000 watcher=hconnection-0xba8d91c0x0, quorum=en01.com:2181, baseZNode=/hbase
17/08/04 09:18:34 INFO zookeeper.ClientCnxn: Opening socket connection to server en01.com/10.1.20.30:2181. Will not attempt to authenticate using SASL (unknown error)
17/08/04 09:18:34 INFO zookeeper.ClientCnxn: Socket connection established to en01.com/10.1.20.30:2181, initiating session
17/08/04 09:18:34 INFO zookeeper.ClientCnxn: Session establishment complete on server en01.com/10.1.20.30:2181, sessionid = 0x65da444b0180e25, negotiated timeout = 60000

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Fri Aug 04 10:01:04 CDT 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68079: row 'tbl_abc,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=dn06.com,60020,1501699598837, seqNum=0

at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:264)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:199)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:56)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:145)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1200)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1109)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:293)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:131)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:56)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:287)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:267)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:139)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:134)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:823)
at com.seagate.dars.imagedefectanalysis.dao.HBaseDao.saveFiles2FS(HBaseDao.java:175)
at com.seagate.dars.imagedefectanalysis.ImageDefectAnalysis.main(ImageDefectAnalysis.java:40)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68079: row 'tbl_defect_analysis_images,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=dn06.com,60020,1501699598837, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:294)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.net.ConnectTimeoutException: 10000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=dn06.com/ip:60020]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:403)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:709)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:880)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:849)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1173)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31751)
at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:176)
at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:155)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 6 more

Announcements