Created on 10-03-2016 05:17 PM
If you've received the error exitCode=7 after enabling Kerberos, you are hitting this Jira bug. Notice the bug outlines the issue but does not outline a solution. The good news is the solution is simple, as I'll document below.
If you've enabled Kerberos through Ambari, you'll get through around 90-95% of the last step "Start and Test Services" and then receive the error:
16/09/26 23:42:49 INFO mapreduce.Job: Running job: job_1474928865338_0022 16/09/26 23:42:55 INFO mapreduce.Job: Job job_1474928865338_0022 running in uber mode : false 16/09/26 23:42:55 INFO mapreduce.Job: map 0% reduce 0% 16/09/26 23:42:55 INFO mapreduce.Job: Job job_1474928865338_0022 failed with state FAILED due to: Application application_1474928865338_0022 failed 2 times due to AM Container for appattempt_1474928865338_0022_000002 exited with exitCode: 7 For more detailed output, check application tracking page: http://master2.fqdn.com:8088/cluster/app/application_1474928865338_0022 Then, click on links to logs of each attempt.Diagnostics: Exception from container-launch. Container id: container_e05_1474928865338_0022_02_000001 Exit code: 7 Stack trace: ExitCodeException exitCode=7: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Shell output: main : command provided 1 main : run as user is ambari-qa main : requested yarn user is ambari-qa Container exited with a non-zero exit code 7 Failing this attempt. Failing the application.
You'll notice running "Service Checks" for Tez, MapReduce2, YARN, Pig (any service that involves creating a YARN container) will fail with the exitCode=7. This is because in YARN, the local-dirs likely has the "noexec" flag specified meaning the binaries that are added to these directories cannot be executed.
Open /etc/fstab (with the proper permissions) and remove the noexec flag under all mounted drives specified under "local-dirs" in YARN. Then either remount or reboot your machine - problem solved.