Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while running insert statement on hive

Error while running insert statement on hive

New Contributor

hi,

I'm trying to execute insert query on hive tables from query editor.I get below error in error log. Please help. Mentioned some settings and properties ,might be helpful for you to understand. I'm using HDP 2.4.2 with 4 datanodes (each with 5 TB storage and 64GB RAM) and 1 namenode.

tez container size = 4096 MB

number of containers held = 3

Hive.auto.convert.join.noconditionaltask.size=1512MB

metastore heap size = 8192MB

tez.am.resource.memory.mb = 8192MB

tez.task.resource.memory.mb = 4096 MB

hive.tez.java.opts = 3274

mapreduce.task.io.sort.mb =2047MB

Error::

INFO : Session is already open INFO : Dag name: insert into evigil_firewall_details_um...10(Stage-1) INFO : Tez session was closed. Reopening... INFO : Session re-established. INFO : INFO : Status: Running (Executing on YARN cluster with App id application_1484654153148_0008) INFO : Map 1: -/-Reducer 2: 0/1Reducer 3: 0/24 INFO : Map 1: 0/1Reducer 2: 0/1Reducer 3: 0/24 INFO : Map 1: 0(+0,-1)/1Reducer 2: 0/1Reducer 3: 0/24 INFO : Map 1: 0(+0,-2)/1Reducer 2: 0/1Reducer 3: 0/24 INFO : Map 1: 0(+0,-3)/1Reducer 2: 0/1Reducer 3: 0/24 ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1484654153148_0008_1_00, diagnostics=[Task failed, taskId=task_1484654153148_0008_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_e23_1484654153148_0008_01_000002 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_e23_1484654153148_0008_01_000002 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]], TaskAttempt 1 failed, info=[Container container_e23_1484654153148_0008_01_000003 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_e23_1484654153148_0008_01_000003 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]], TaskAttempt 2 failed, info=[Container container_e23_1484654153148_0008_01_000004 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_e23_1484654153148_0008_01_000004 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]], TaskAttempt 3 failed, info=[Container container_e23_1484654153148_0008_01_000005 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_e23_1484654153148_0008_01_000005 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1484654153148_0008_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE] ERROR : Vertex killed, vertexName=Reducer 3, vertexId=vertex_1484654153148_0008_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:24, Vertex vertex_1484654153148_0008_1_02 [Reducer 3] killed/failed due to:OTHER_VERTEX_FAILURE] ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1484654153148_0008_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1484654153148_0008_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE] ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:2

2 REPLIES 2
Highlighted

Re: Error while running insert statement on hive

New Contributor

@Mahendra Dahiya

Could you post the values of the following configuration settings:

  • tez.task.launch.cmd-opts
  • tez.am.launch.cmd-opts
  • hive.tez.java.opts

My suspicion is that they may contain conflicting Garbage Collection Options or a mix of options that aren't compatible (i.e. CMS GC options mixed with G1GC options)

Can you also attach your yarn application logs?

yarn logs -applicationId application_1484654153148_0008 > /tmp/application_1484654153148_0008.log

Re: Error while running insert statement on hive

Mentor

@Mahendra Dahiya

Adding jar to tez.aux.uris should resolve the issue