This error was called during the execution of the job controller within the MapReduce job. Here's a similar one with the same root problem.       Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/orc/OrcNewOutputFormat at com.who.bgt.logloader.schema.OrcFileLoader.run(OrcFileLoader.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at com.who.bgt.logloader.schema.OrcFileLoader.main(OrcFileLoader.java:45) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.orc.OrcNewOutputFormat at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 8 more   The specific line it is complaining about is here:   job.setOutputFormatClass(OrcNewOutputFormat.class);   The obvious problem is that it's failing to find the OrcNewOutputFormat class definition, which is in hive-exec-0.13.1-cdh5.3.5.jar   I pushed the jar to hdfs://lib/hive-exec..., and within my main function, I call the following before I run the job:   DistributedCache.addFileToClassPath(new Path("/lib/hive-exec-0.13.1-cdh5.3.5.jar"), lConfig); Can you be more explicit on how I go about making sure my distributed-cache configs work?   Optimally, I shouldn't have to stuff this one in the distributed cache since it sits in /opt/cloudera/parcels/CDH-5.3.5-1.cdh5.3.5.p0.4/jars/hive-exec-0.13.1-cdh5.3.5.jar on all of my slave nodes, but I also can't figure out how to tell MapReduce to look there.   ... View more

Thank you, mfox. My problem was that the basic install set all of HDFS's groups to "superuser" instead of "hadoop". Changing it to "hadoop" allowed mapreduce to write its history logs to the correct location. ... View more

I think that the "drilling down into the individual map/reduce tasks" is where this falls apart for me.   When I click on the task (e.g. application_1431658373269_0170) it shows me a list of application masters.    From there I can click on the node id (e.g. hadoopslave0011p1mdw1.sendgrid.net:8042) This takes me to a page where I can see all of the containers currently running, and see logs for the node itself, which isn't what I need.     I can also click on "logs" for the application master. This takes me to a page that says   Error getting logs for container_e23_1431658373269_0170_01_000001   Which tells me that the cluster is mis-configured in some way, and isn't even producing them. Can you recommend a next step? ... View more

I'm on Cloudera 5.3.3. Here's my command line and output:   [hadoop]$ hdfs dfs -du / 2298676940886 6896030822658 /output 21297905593 63893716779 /tmp 6072184915396 18216555409976 /user ... View more

That was it! Thank you! ... View more