08-04-2018 09:58 AM
Thank you for the update.
As I understand the logs doesn't contain messages on failure to re-acquire authn token, and authn tokens were successfully re-acquired some time before the error referenced in the first post.
Two more questions:
I0503 09:29:01.528237 42404 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to <Kudu_leader_IP>:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired
I might be wrong, but those messages about expired authn tokens might be a red herring.
08-06-2018 05:49 AM
Unfortunately, the logrotation policies have just removed the logs for the case in the first post.
Are there any ideas/suggestions based on difference in table types - external and not? Both were based on Kudu, but external continued to work properly.
09-17-2018 11:14 AM
I have an update on this issue.
I've got a chance to look at this issue closer having on hand Impalad's log. As it turned out, there is a bug in the Kudu C++ client and I think your case was a manifestation of that bug as well:
I posted a patch to fix the issue and I hope the fix will be reviewed and merged soon.
09-18-2018 03:01 AM
Great news, thanks!
Since it is Kudu client, does it mean it will be available when a new version of Impala is released with fixed Kudu client?
09-19-2018 01:18 PM
Right, to make the fix into Impala it's necessary to relink impalad with patched Kudu client.
impalad is linked against the kudu_client dynamically, so in theory it might be possible just to replace the libkudu_client.so.0 library with the patched version. However, that's really messy and I would not recommend that. If you use CDH anyway, the best option is to wait for next release of CDH. I don't know what version that will be, though.
If you want a workaround, set the --authn_token_validity_seconds flag to be months or even one year long (i.e. --authn_token_validity_seconds=31536000) and restart Kudu masters. You will need to enable experimental flags as well (i.e. add --unlock_experimental_flags).