Hello everyone. I need some help.
I am currently working on a Proof of Concept (POC) for building a platform for a client, which is why I am unable to open a case and am asking my question here in this community.
We are using spark3 and kudu, and there seems to be a problem. MIT Kerberos is set up, and I'm working in a CM7.10.1, CDP7.1.7sp2 environment.
It's a Spark On YARN environment, and I'm executing commands like this:
spark-submit --master yarn --deploy-mode cluster \--keytab /etc/security/keytabs/user1.keytab \
--principal user1@example.com \
--jars /opt/cloudera/parcels/CDH/lib/kudu/kudu-spark3_2.12.jar \....
The spark application runs normally on YARN and operates, but an error occurs when the Kerberos ticket is renewed after 7 days. Since the spark application restarts once by default, I think it will be completely killed after 14 days, but for now, it's still running after restarting.
Spark error
org.apache.kudu.client.NonRecoverableException: cannot re-acquire authentication token after 5 attempts (couldn't find a valid master in (m01.example.com:7051, m02.example.com:7051, m03.example.com:7051) exception
Kudu master warning
Unauthorized connection attempt: Server Connection negotiation failed: Server connection from 123.123.123.123:42026: token expired.
Failed RPC negotiation trace: {
.....
Negotiation complete: network error: Server connection negotiation failed: Server connection from 123.123.123.123:42026: BlockingRecv error: recv of EOF from 123.123.123.123:42026 (error 108)
I don't see any warnings or issues in the krb5kdc.log. It seems like the related error was resolved in Apache Kudu version 1.5, but I am using CDP7.1.7sp2, Apache kudu 1.15 version. Please help me. Thank you in advance.
