Reply
Cloudera Employee
Posts: 19
Registered: ‎03-16-2017

Re: Authentication token expired at impala coordinators after Kudu restart

Thank you for the update.

 

As I understand the logs doesn't contain messages on failure to re-acquire authn token, and authn tokens were successfully re-acquired some time before the error referenced in the first post.

 

Two more questions:

  1. How far the timestamp referenced in the very fist post was from the time when Impala was unable to launch a query?
    I0503 09:29:01.528237 42404 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to <Kudu_leader_IP>:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired
  2. What was the error message output to the user (if any), when Impala was unable to launch the query?

 

I might be wrong, but those messages about expired authn tokens might be a red herring.

 

Thank you!

Explorer
Posts: 19
Registered: ‎05-03-2018

Re: Authentication token expired at impala coordinators after Kudu restart

Unfortunately, the logrotation policies have just removed the logs for the case in the first post.

 

Are there any ideas/suggestions based on difference in table types - external and not? Both were based on Kudu, but external continued to work properly.

 

Cloudera Employee
Posts: 19
Registered: ‎03-16-2017

Re: Authentication token expired at impala coordinators after Kudu restart

Hi,

 

I have an update on this issue.

 

I've got a chance to look at this issue closer having on hand Impalad's log.  As it turned out, there is a bug in the Kudu C++ client and I think your case was a manifestation of that bug as well: 

 

  https://issues.apache.org/jira/browse/KUDU-2580

 

I posted a patch to fix the issue and I hope the fix will be reviewed and merged soon.

 

 

Best regards,

 

Alexey

 

Explorer
Posts: 19
Registered: ‎05-03-2018

Re: Authentication token expired at impala coordinators after Kudu restart

Great news, thanks!

Since it is Kudu client, does it mean it will be available when a new version of Impala is released with fixed Kudu client?

Cloudera Employee
Posts: 19
Registered: ‎03-16-2017

Re: Authentication token expired at impala coordinators after Kudu restart

Right, to make the fix into Impala it's necessary to relink impalad with patched Kudu client.

 

impalad is linked against the kudu_client dynamically, so in theory it might be possible just to replace the libkudu_client.so.0 library with the patched version.  However, that's really messy and I would not recommend that.  If you use CDH anyway, the best option is to wait for next release of CDH.  I don't know what version that will be, though.

 

If you want a workaround, set the --authn_token_validity_seconds flag to be months or even one year long  (i.e. --authn_token_validity_seconds=31536000) and restart Kudu masters.  You will need to enable experimental flags as well (i.e. add --unlock_experimental_flags).

Announcements