Reply
Highlighted
New Contributor
Posts: 5
Registered: ‎08-15-2016
Accepted Solution

Kudu TLS Error

Hello,

We are running into an error with an external Java Client connecting through the Java API to connect to Kudu. On the server side we see a TLS connection being attempted:

W0616 13:01:28.007700 28840 negotiation.cc:290] Failed RPC negotiation. Trace:
0616 13:01:27.987227 (+     0us) reactor.cc:373] Submitting negotiation task for server connection from 10.x.x.xxx:64686
0616 13:01:27.987396 (+   169us) server_negotiation.cc:118] Beginning negotiation
0616 13:01:27.987397 (+     1us) server_negotiation.cc:282] Waiting for connection header
0616 13:01:27.987764 (+   367us) server_negotiation.cc:290] Connection header received
0616 13:01:27.988083 (+   319us) server_negotiation.cc:246] Received NEGOTIATE NegotiatePB request
0616 13:01:27.988084 (+     1us) server_negotiation.cc:331] Received NEGOTIATE request from client
0616 13:01:27.988098 (+    14us) server_negotiation.cc:258] Sending NEGOTIATE NegotiatePB response
0616 13:01:27.988120 (+    22us) server_negotiation.cc:139] Negotiated authn=SASL
0616 13:01:27.998217 (+ 10097us) server_negotiation.cc:246] Received TLS_HANDSHAKE NegotiatePB request
0616 13:01:27.998402 (+   185us) server_negotiation.cc:258] Sending TLS_HANDSHAKE NegotiatePB response
0616 13:01:28.001852 (+  3450us) server_negotiation.cc:246] Received TLS_HANDSHAKE NegotiatePB request
0616 13:01:28.005023 (+  3171us) server_negotiation.cc:258] Sending TLS_HANDSHAKE NegotiatePB response
0616 13:01:28.005048 (+    25us) server_negotiation.cc:496] Negotiated TLSv1.2 with cipher suite AES128-SHA256
0616 13:01:28.007617 (+  2569us) negotiation.cc:281] Negotiation complete: Network error: Server connection negotiation failed: server connection from 10.x.x.xxx:64686: BlockingRecv error: failed to read from TLS socket: Cannot send after transport endpoint shutdown (error 108)
Metrics: {"negotiator.queue_time_us":139,"thread_start_us":119,"threads_started":1}

The code we are using to connect is pretty straight forward:

KuduClient client = new KuduClient.KuduClientBuilder(KUDU_MASTER).build();

if (!client.getTablesList(tableName).getTablesList().isEmpty()){
System.out.println("deleting table if exists...");
client.deleteTable(tableName);
}

it fails in "if (!client.get....)" line

 

This is the stack trace from the Java app:

 

Exception in thread "main" org.apache.kudu.client.NoLeaderFoundException: Master config (kudumaster.xxxx.xxx:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: [Peer master-kudumaster.xxxx.xxx:7051] Connection disconnected
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:241)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:47)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:309)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:298)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:238)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:292)
at org.apache.kudu.client.TabletClient.failOrRetryRpc(TabletClient.java:691)
at org.apache.kudu.client.TabletClient.failOrRetryRpcs(TabletClient.java:668)
at org.apache.kudu.client.TabletClient.cleanup(TabletClient.java:657)
at org.apache.kudu.client.TabletClient.channelDisconnected(TabletClient.java:608)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.apache.kudu.client.TabletClient.handleUpstream(TabletClient.java:601)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at

The cluster is kerberized with TLS/SSL enabled and has 3 master kudu servers with a handful of tablet servers. The same code is working fine on a non-kerberized cluster. 

 

 

Any help would be greatly appreciated!

Posts: 614
Topics: 3
Kudos: 93
Solutions: 61
Registered: ‎08-16-2016

Re: Kudu TLS Error

I would try loading the CA cert that signed the certs for the Kudu master and worker nodes into the JSSE truststore. Do this on the host you are running the Kudu app on. Then any Java applications will use it when negotiating TLS handshakes. You could also create a truststore and use that when launching the Kudu app.

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_sg_create_key_trust.html
Cloudera Employee J-D
Cloudera Employee
Posts: 53
Registered: ‎07-08-2013

Re: Kudu TLS Error

This question has been posted in many different places and it seems to be related to the fact that this is run from a Windows host which has never been tested. It's not clear it's really a TLS issue.

 


mbigelow wrote:
I would try loading the CA cert that signed the certs for the Kudu master and worker nodes into the JSSE truststore. Do this on the host you are running the Kudu app on. Then any Java applications will use it when negotiating TLS handshakes. You could also create a truststore and use that when launching the Kudu app.

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_sg_create_key_trust.html

That's not how it works in Kudu though, the CA cert is stored in the Kudu master, no messing with JSSE, see http://kudu.apache.org/docs/security.html

New Contributor
Posts: 5
Registered: ‎08-15-2016

Re: Kudu TLS Error

Thanks the responses back! J-D did you have an example post I can take a look at being that it could be since the Java client is on a Windows machine?

Announcements