nodemanger everyday down at 00.00

[#1]

Hello!

all nodemanager down everyday withou FATAL, ERROR in logs

Its all log 

2024-06-28 00:11:50,491 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46342:appattempt_1719446422443_24818_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)

2024-06-28 00:11:50,492 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]

2024-06-28 00:11:50,748 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46366:appattempt_1719446422443_24827_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)

2024-06-28 00:11:50,748 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]

2024-06-28 00:11:50,904 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46380:appattempt_1719446422443_24824_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)

2024-06-28 00:11:50,904 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]

2024-06-28 00:11:50,974 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46388:appattempt_1719446422443_24815_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)

2024-06-28 00:11:50,974 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]

2024-06-28 00:11:51,043 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46394:appattempt_1719446422443_24816_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)

2024-06-28 00:11:51,043 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]

2024-06-28 00:11:51,405 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46436:appattempt_1719446422443_0836_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)

2024-06-28 00:11:51,405 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]

2024-06-28 00:11:51,558 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46450:appattempt_1719446422443_24819_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)

2024-06-28 00:11:51,558 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]

2024-06-28 00:11:52,104 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46506:appattempt_1719446422443_24379_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)

2024-06-28 00:11:52,104 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]

[#2 · Accepted solution]

Please check if you have tgt created for all the Nodemanagers. It will fix the issue.