For my case, I observed the spark job was working fine on some hosts and hitting the above exception for a couple of worker hosts.   Found that the issue with spark-submit --version on hosts. working hosts spark-submit version was version 2.4.7.7.1.7.0-551 and non-working hosts spark-submit version was version 3.1.2   I created the symbolic link with the correct spark-submit version file and the issue got resolved.   ``` [root@host bin]# cd /usr/local/bin [root@hostbin]# ln -s /etc/alternatives/spark-submit spark-submit ```     ... View more