Spark3 Issue: Executor is not able to register with the External Shuffle service : Unknown message type: -22

[#1]

HI Guys, 

I am having an issue running an Spark 3 jobs: 

Driver log: 
ERROR: cluster:YarnClusterScheduler: [dispatcher-CoarseGrainedScheduler] : Lost executor 1 on host_name: Unable to create executor due to unable to register with external shuffle service due to: java.lang.IllegalException: Unknown message type: -22

Executor log: 
ERROR: Exception while bootstrapping client in 3 ms. 
Java.lang.RuntimeException: java.lang.IllegalException: Unknown message type: -22

Executor making 3 attempts with 5 second waiting period, then exiting. 

From my research, I found recommendation on increasing the shuffle register timeout. Any recommendation is appreciated. 

[#2]

Can you try disabling the spark.authentication ? 

spark.authenticate = false