Thank you for the update.
As I understand the logs doesn't contain messages on failure to re-acquire authn token, and authn tokens were successfully re-acquired some time before the error referenced in the first post.
Two more questions:
I0503 09:29:01.528237 42404 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to <Kudu_leader_IP>:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired
I might be wrong, but those messages about expired authn tokens might be a red herring.
Unfortunately, the logrotation policies have just removed the logs for the case in the first post.
Are there any ideas/suggestions based on difference in table types - external and not? Both were based on Kudu, but external continued to work properly.
I have an update on this issue.
I've got a chance to look at this issue closer having on hand Impalad's log. As it turned out, there is a bug in the Kudu C++ client and I think your case was a manifestation of that bug as well:
I posted a patch to fix the issue and I hope the fix will be reviewed and merged soon.
Great news, thanks!
Since it is Kudu client, does it mean it will be available when a new version of Impala is released with fixed Kudu client?
Right, to make the fix into Impala it's necessary to relink impalad with patched Kudu client.
impalad is linked against the kudu_client dynamically, so in theory it might be possible just to replace the libkudu_client.so.0 library with the patched version. However, that's really messy and I would not recommend that. If you use CDH anyway, the best option is to wait for next release of CDH. I don't know what version that will be, though.
If you want a workaround, set the --authn_token_validity_seconds flag to be months or even one year long (i.e. --authn_token_validity_seconds=31536000) and restart Kudu masters. You will need to enable experimental flags as well (i.e. add --unlock_experimental_flags).