Hi Sara, I run vulnerability scans and our scanner picking up Spark 1.6 banner from the following path- for CVE-2018-8024 vulnerability, you did mention this vulnerability doesn't affect SPark 1.6 but didn't give detail reasons. This is where Qualys picksup the banner- /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/spark/conf/spark-env.sh: line 75: /usr/appl/cloudera/java/jdk1.8.0_162: is a directory Welcome to ____              __ / __/__  ___ _____/ /__ _\ \/ _ \/ _ `/ __/  '_/ /___/ .__/\_,_/_/ /_/\_\   version 1.6.0 /_/   Also we have 2 versions of SPARK running- do you really need version 1.6 to run v 2.3.0 Can you please help, advise.   ... View more

Without much context, you should go to YARN --> Resource Manager web UI, find the failed job corresponding to the distcp, and drill into it to find the failed reduce task. You should be able to find out more there in the log. ... View more