Vertex failed, vertexName=Map 1, vertexId=vertex_1486082581831_0016_1_00, diagnostics ........ Container failed, exitCode=1. Exception from container-launch

[#1]

Hi,

When implementing Ranger-hive policy , i made below changes

1) hive.conf.restricted.list

old : hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role

new : hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role,hive.security.authorization.enabled

2) hive.tez.container.size

old : -1

New : 2048 mb ( equal to yarn.scheduler.minimum-allocation-mb)

3) hive.auto.convert.join.noconditionaltask.size

old : 52428800

new : 357913941

4) hive.security.authorization.enabled

old : false

new : true

5) hive.security.authorization.manager :

old : org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory

new : org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory

There after started getting below error from both beeline and hive , when i execute commands

Status: Failed

Vertex failed, vertexName=Map 1, vertexId=vertex_1486082581831_0016_1_00, diagnostics=[Task failed, taskId=task_1486082581831_0016_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_1486082581831_0016_01_000002 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch.

Container id: container_1486082581831_0016_01_000002

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)

at org.apache.hadoop.util.Shell.run(Shell.java:487)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

]], TaskAttempt 1 failed, info=[Container container_1486082581831_0016_01_000003 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch.

Container id: container_1486082581831_0016_01_000003

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)

at org.apache.hadoop.util.Shell.run(Shell.java:487)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

***************

yarn version is 2.7.1.2.4

tez version is 0.7.0.2.4

[#2]

I can't parse for sure what the error is coming from; however, I did notice you set your container size to 2048. I start with 8048 and 7050 for heap. can you try this and rerun?

[#3]

Hi Sunile,

I cannot increase hive.tez.container.size more than yarn.scheduler.maximum-allocation-mb . Increase to 4000mb , and still getting the same error.

below are my setting now.

yarn.scheduler.minimum-allocation-mb : 2048mb

yarn.scheduler.maximum-allocation-mb : 5120mb

hive.tez.container.size 2048 ----> 4000mb

[root@node01 ~]# vmstat -s 8

009212 K total memory

3091192 K used memory

3869668 K active memory

2283236 K inactive memory

723736 K free memory

[#4]

sunile , after i set hive.tez.container.size to -1 , this issue got resolved.

But still i get below error when in run insert statements from beeline

17/02/07 04:06:36 [HiveServer2-Background-Pool: Thread-137]: WARN tez.DagUtils: hive.tez.java.opts will be ignored because hive.tez.container.size is not set!