Invalid/conflicting GC options found error in Hive

[#1]

Hi All!

I'm running query at hive and getting the following error. Please help!

hive> select serialno, count(*) from cdr_data where hday='2017-08-01' group by serialno having count(*) >1;

Query ID = hdfs_20170816104340_764cc276-8015-4655-b153-a6bcf445a603 Total jobs = 1 Launching Job 1 out of 1 Dag submit failed due to Invalid TaskLaunchCmdOpts defined for Vertex Map 1 : Invalid/conflicting GC options found, cmdOpts="-server -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.0.3-8 -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/ -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA " stack trace: [org.apache.tez.dag.api.DAG.createDag(DAG.java:879), org.apache.tez.client.TezClientUtils.prepareAndCreateDAGPlan(TezClientUtils.java:678), org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:444), org.apache.tez.client.TezClient.submitDAG(TezClient.java:411), org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:470), org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:188), org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160), org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89), org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1748), org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1494), org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1291), org.apache.hadoop.hive.ql.Driver.run(Driver.java:1158), org.apache.hadoop.hive.ql.Driver.run(Driver.java:1148), org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:217), org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169), org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380), org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:740), org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685), org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625), sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method), sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:498), org.apache.hadoop.util.RunJar.run(RunJar.java:233), org.apache.hadoop.util.RunJar.main(RunJar.java:148)] retrying... FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

[#2 · Accepted solution]

@Gulshan Agivetova

Please check and share the values of the following parameters "tez.am.launch.cmd-opts" and "hive.tez.java.opts" they should not be conflicting. Specially the GC options.

Try this: You should get rid of the "-XX:+UseParallelGC" option (remove this option) from the "tez.am.launch.cmd-opts" and "hive.tez.java.opts" properties at the very first place and then restart it.

This is because "-XX:+UseG1GC and -XX:+UseParallelGC" Should never be used together."

[#3]

@Gulshan Agivetova

Good to know that setting the -XX:+UseG1GC for hive.tez.java.opts worked.

- It will be really great if you can mark this thread as "Accepted" (Answered) so that it will be very useful for HCC users to quickly browse for the solutions.

[#4]

@Jay SenSharma Thank you very much! 🙂

It really helped! I've set -XX:+UseG1GC for hive.tez.java.opts and it works!

Thank you!