Hi,   we have a cluster with 8 nodes on CDH5 (5.0.2) with Yarn MRv2 in use and a big problem which is probably due to the Config.   In addition to Hadoop, we also use Imapala so we can not use give all ressoures to yarn. Each of our nodes have 128GB of RAM and 12 cores.   Currently sees the Memory config for Yarn as follows:   mapreduce.map.memory.mb = 8Gib mapreduce.reduce.memory.mb = 8Gib yarn.app.mapreduce.am.resource.mb = 8Gib mapreduce.map.java.opts.max.heap = 6960MiB mapreduce.reduce.java.opts.max.heap = 6960MiB "Java Heap Size in bytes of NodeManager" = 8Gib yarn.nodemanager.resource.memory-mb = 80Gib   Now we get the problem that if we run multiple applications in parallel all stop and no one finished. it looks as if they hang forever. I see no exception or errors in "/var/log/hadoop-yarn" (Debug Log Level).   I would be glad if someone can help? 🙂   BG ... View more

Hi,   i want to create an external hive tebale mapped on hbase in my oozie workflow. on cdh4 it works without problems but using the script on cdh5 i got an "java.lang.NoClassDefFoundError" exception. I have added all jars like in the documentation:   http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_hive_hbase.html   I also added the jars with the "hive.aux.jars.path" property at the hive-site.xml   Hive script:   CREATE EXTERNAL TABLE IF NOT EXISTS hive_table( hash BIGINT, f1 STRING )STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( 'hbase.columns.mapping' =':key,cf1:f1')TBLPROPERTIES('hbase.table.name' = 'hbase_table');   Error:   3029 [main] ERROR hive.ql.exec.DDLTask - java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/mapreduce/TableInputFormatBase at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:305) at org.apache.hadoop.hive.ql.metadata.Table.<init>(Table.java:98) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:991) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:894) at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:3308) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:278) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1263) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:921) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:357) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:455) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:465) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:746) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:315) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:276) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 60 more   My Cluster is configured with CDH5.0.2 and CDM5 with Yarn MRv2     I hope you can help me which jar file is missing ... View more