After executing TestDFSIO command: sudo -u hdfs /usr/hdp/current/hadoop-client/bin/hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 10 -filesize 40   I get the following error java.lang.NoClassDefFoundError: junit/framework/TestCase at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.test.MapredTestDriver.<init>(MapredTestDriver.java:109) at org.apache.hadoop.test.MapredTestDriver.<init>(MapredTestDriver.java:61) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:147) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 21 more 19/12/09 17:11:26 INFO fs.TestDFSIO: TestDFSIO.1.8   On another installation of HDP v 2.5 works like a charm.   Any ideas why? ... View more

I'm using HDP-3.1 (spark2.3, python 3.6, Centos7)   I was able to configure it to work with python2.7, but with 3.6 I got the following error when running a notebook: Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.   I have python 3.6 installed on all of the workers and environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set (see my configuration bellow):     Am I missing something? ... View more