Member since
05-26-2016
19
Posts
0
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 28991 | 06-16-2016 05:47 AM | |
| 29040 | 06-10-2016 06:02 AM |
04-03-2018
09:55 AM
Hi csguna, thanks for your reply. Yes indeed we have a job which connects to Hive, retrieves some data, then closes the connection. This is a finite job, with a start and stop within a few hours. Generally, 2-4 connections are made during the job's lifetime, where we connect, retrieve some data, then disconnect, or connect, insert some data via LOAD DATA INPATH, then disconnect. This job fails frequently in environments with higher amounts of traffic/activity. I believe there are two HS2 instances in that cluster. hive.server2.session.check.interval=1 hour hive.server2.idle.operation.timeout=1 day hive.server2.idle.session.timeout=1 day What worries me is that our session appears to just be closed with no reason as we're just trying to connect. Inspecting the driver's code: Variant localVariant3 = getOptionalSetting("SocketTimeOut", paramConnSettingRequestMap);
.......
if ((null == localVariant3) && (0 != m_settings.m_timeout)) { m_hiveClient.closeSession(); in com.cloudera.hiveserver2.hivecommon.core.HiveJDBCCommonConnection. I'm not following the logic behind the code which simply examines the value of SocketTimeout and if it happens to not be specified, then it closes the session. Why? And why would that fail with a "Read timed out"? In theory, we can try running with the SocketTimeout set to 0 tacked onto the JDBC URL we're using. But I'd really want to understand a) whether that's necessary, b) if so, then why, and c) is there something else that might be going on.
... View more
03-28-2018
05:39 AM
Hi, We've been seeing a frequent occurrence of the types of errors I'm going to quote below. We are using the Cloudera Hive JDBC driver version 2.5.18. (The 2.5.20 version is out there, we've not upgraded to, but doesn't seem to have any fixes related to the issues below). CDH version = 5.9.1. Does anyone have any idea as to why we may be frequently seeing these errors? The Hive URL we're using looks like this (one string but breaking down property by property for readability): jdbc:hive2://<host>:10000/default;
AuthMech=1;
KrbRealm=<realm>;
KrbHostFQDN=<host fqn>;
KrbServiceName=hive;
UseNativeQuery=1;
ssl=1;
sslTrustStore=<store path>;
trustStorePassword=<store pwd> The errors below tend to happen when we try to obtain a connection. Presumably, the SocketTimeout driver configuration option would not be relevant in this case (have considered setting it to 0 so that idle connections are not closed). The errors: Read timed out org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:182)
at com.cloudera.hiveserver2.hivecommon.api.HS2ClientWrapper.recv_CloseSession(Unknown Source)
at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:169)
at com.cloudera.hiveserver2.hivecommon.api.HS2ClientWrapper.CloseSession(Unknown Source)
at com.cloudera.hiveserver2.hivecommon.api.HS2Client.closeSession(Unknown Source)
at com.cloudera.hiveserver2.hivecommon.core.HiveJDBCCommonConnection.connect(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.AbstractDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) Out of sequence response org.apache.thrift.TApplicationException: CloseSession failed: out of sequence response
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:182)
at com.cloudera.hiveserver2.hivecommon.api.HS2ClientWrapper.recv_CloseSession(Unknown Source)
at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:169)
at com.cloudera.hiveserver2.hivecommon.api.HS2ClientWrapper.CloseSession(Unknown Source)
at com.cloudera.hiveserver2.hivecommon.api.HS2Client.closeSession(Unknown Source)
at com.cloudera.hiveserver2.hivecommon.core.HiveJDBCCommonConnection.connect(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.AbstractDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270) Close session error java.sql.SQLException: [Cloudera][HiveJDBCDriver](500151) Error setting/closing session: Close Session Error.
at com.cloudera.hiveserver2.hivecommon.api.HS2Client.closeSession(Unknown Source) Any insight would be appreciated.
... View more
Labels:
- Labels:
-
Apache Hive
03-28-2018
05:03 AM
A while back, I made a comment to the effect of "We may have to munge these files before we can use them" 🙂 For those of you giggling about the use of the word munge, it's widely used on the East Coast to mean "edit" or "overwrite". Typically files are not very edible and XML would not taste good 🙂 🙂 Cheers!
... View more
03-28-2018
05:01 AM
Have you tried setting dfs.namenode.kerberos.principal?
... View more
06-16-2016
05:47 AM
Because of our requirements (to be able to be targeted toward a different cluster based on deployment, and the HDFS config files potentially having cluster-internal host names in them), we're going with the approach of maintaining the minimalistic set of Configuration properties required to make Kerberos work on the client side. These are, again: * dfs.namenode.kerberos.principal * hadoop.rpc.protection Having said that, Harsh's comments are all valid and relevant.
... View more
06-10-2016
06:02 AM
Thanks, Harsh, very helpful. I've been poking around on an edge node, so I do have access to hdfs-site.xml and core-site.xml. We may have to munge these files before we can use them, as they contain some values such as host names for fs.defaultFS which are cluster internal; we'll have to use different host names to be able to get in from outside the cluster... Since we deal with multiple clusters organized by stage (dev, prod, etc.), we'd have to maintain multiple pairs of core-site.xml, hdfs-site.xml files, and load them dynamically at runtime via Configuration.addResource() method...
... View more
06-09-2016
06:56 PM
Harsh, There are 3 host names at play: A, B, and C. Things have started working, actually, when I set fs.defaultFS to one of these (B); originally I was using A. I'm told, however, that all 3 are supposed to be 'active'. >> you are passing only a single hostname for the NN Per this comment you made, should I be passing in all 3 hostnames? If so, how? The doc states that fs.defaultFS is "The name of the default file system." so a) should all 3 names be passed and b) if so, how? Thanks for your help.
... View more
06-09-2016
06:25 AM
Harsh, Thanks for the hadoop-client suggestion, I've changed the pom file. That did not make any difference, however, as far as the issue is concerned. As far as downloading a client configuration zip, is that something I could do via Hue? I do not have access to the main SCM interface. Any other means of retrieving this? Per your comment "You certainly do need to set hadoop.rpc.protection to the exact value the cluster expects", I've tried the other values. The "authentication" and "integrity" did not make a difference, I was still getting the error “No common protection layer between client and server”. However, setting "hadoop.rpc.protection" to "privacy" caused a different type of error (see below). Any recommendations at this point? Thanks. Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1775) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1402) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4221) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:881) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getFileInfo(AuthorizationProviderProxyClientProtocol.java:526) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:822) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
... View more
06-08-2016
08:10 PM
Hi Harsh, My Hadoop dependencies are: hadoop-common, hadoop-hdfs, version=2.5.0, since we're running CDH 5.3. Does that sound like the right version? <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.5.0</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.5.0</version> </dependency> Thanks. - Dmitry
... View more
06-08-2016
04:45 PM
I'm trying to communicate programmatically to a Hadoop cluster which is kerberized (CDH 5.3/HDFS 2.5.0). I have a valid Kerberos token on the client side. But I'm getting an error as below, "No common protection layer between client and server". What does this error mean and are there any ways to fix or work around it? Is this something related to HDFS-5688? The ticket seems to imply that the property "hadoop.rpc.protection" must be set, presumably to "authentication" (also per e.g. this). Would this need to be set on all servers in the cluster and then the cluster bounced? I don't have easy access to the cluster so I need to understand whether 'hadoop.rpc.protection' is the actual cause. It seems that 'authentication' should be the value used by default, at least according to the core-default.xml documentation. java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for principal1/server1.acme.net@xxx.acme.net to server2.acme.net/10.XX.XXX.XXX:8020; Host Details : local host is: “some-host.acme.net/168.XX.XXX.XX”; destination host is: “server2.acme.net”:8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)
... 11 more Caused by: java.io.IOException: Couldn't setup connection for principal1/server1.acme.net@xxx.acme.net to server2.acme.net/10.XX.XXX.XXX:8020; at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:671)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
at org.apache.hadoop.ipc.Client.call(Client.java:1382)
... 31 more Caused by: javax.security.sasl.SaslException: No common protection layer between client and server at com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:251)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:186)
at org.apache.hadoop.security.SaslRpcClient.saslEvaluateToken(SaslRpcClient.java:483)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:427)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:552)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:717)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
... 34 more
... View more
Labels: