Skip to main content
Dombai_Gabor
Dombai_Gabor
Silver ⭐️⭐️
February 28, 2020
Solved

YARN NodeManagers failed to start with permission issue after Kerberizaton in Cloudera Runtime 7.0.3

Dear Community,

 

After a successful installion of Cloudera Runtime 7.0.3, we tried to do a Kerberization process. (We did the same before with 5.14)

Everything went fine with the Kerberos wizard, but after in the config deployment phase, the YARN NodeManagers failed to start with the following error:

 

Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:394)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:936)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1016)
Caused by: java.io.IOException: Linux container executor not configured properly (error=-1)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:307)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:392)
... 3 more
Caused by: org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: java.io.IOException: Cannot run program "/var/lib/yarn-ce/bin/container-executor": error=13, Permission denied
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:183)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:206)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:300)
... 4 more
Caused by: java.io.IOException: Cannot run program "/var/lib/yarn-ce/bin/container-executor": error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:938)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
... 6 more
Caused by: java.io.IOException: error=13, Permission denied
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 10 more

 We tried to search the "answer" on the internet. We found that we should add the yarn user in the hadoop group.

 

[root@dwh20-i-cdhwt01 ~]# id yarn
uid=485(yarn) gid=984(yarn) groups=984(yarn),988(hadoop)

 

Nothing happened, the NodeManagers didn't start.

The container-executor script permission:

 

[root@dwh20-i-cdhwt01 ~]# ll /var/lib/yarn-ce/bin/container-executor
---Sr-s--- 1 root yarn 103968 Nov 19 12:34 /var/lib/yarn-ce/bin/container-executor

 

(Just for information, we didn't find container-executor config file on the NodeManagers.)

 

Please somebody help us. 🙂

Thanks.

 

Regards,

Gabor 

 

bgooley
bgooleyAnswer
Gold ⭐️
February 28, 2020

Hi @Dombai_Gabor ,

 

One possible cause of this issue is that the volume is mounted with "noexec".  Since your permissions and group membership seem correct, it is reasonable to check /etc/fstab to see if "noexec" is set where /var/ mounted.

 

Ben

View in thread

13 replies

venkatsambath
Gold ⭐️⭐️⭐️
venkatsambath
Gold ⭐️⭐️⭐️
February 28, 2020

Can you share the result of 

ls -nl /var/lib/yarn-ce/bin/container-executor

This is to make sure the uid of yarn matches the uid on the container-executor

Translate
Dombai_Gabor
Dombai_GaborAuthor
Silver ⭐️⭐️
February 28, 2020

Yes! Here you are. 🙂

 

[root@dwh20-i-cdhwt01 ~]# ls -nl /var/lib/yarn-ce/bin/container-executor
---Sr-s--- 1 0 984 103968 Nov 19 12:34 /var/lib/yarn-ce/bin/container-executor

Translate
bgooley
Gold ⭐️
bgooleyAnswer
Gold ⭐️
February 28, 2020

Hi @Dombai_Gabor ,

 

One possible cause of this issue is that the volume is mounted with "noexec".  Since your permissions and group membership seem correct, it is reasonable to check /etc/fstab to see if "noexec" is set where /var/ mounted.

 

Ben

Translate
    Dombai_Gabor
    Dombai_GaborAuthor
    Silver ⭐️⭐️
    February 28, 2020

    I checked it. Yes, the /var mount point has a noexec parameter. I turned off the cloudera services, edited the fstab, rebooted the hosts, but they dont boot now. 😞 

    Translate
    Dombai_Gabor
    Dombai_GaborAuthor
    Silver ⭐️⭐️
    February 29, 2020

    Ok guys, there are no problems no. 🙂 There was a typo in our fstab file (missing s from the defaults option).

     

    On the other hand, there was a noexec option on the /var mountpoint. After I've removed it, it is working now finally.

    The NodeManagers are working properly.

     

    Thanks guys. 😄

    Translate
    soyoung
    soyoung
    Silver ⭐️
    September 9, 2020

    I resolved this problem by running the 'usermod -G yarn yarn' command.

    Translate
    Hasib1971
    Hasib1971
    Silver ⭐️⭐️
    February 7, 2026

    i have same issue after kerborous enable. But issue not belong to kerborous it is about container-executor can't open : permission denied.  Any better solution you have?

    Translate
    VidyaSargur
    Community Manager
    VidyaSargur
    Community Manager
    February 9, 2026

    @Hasib1971, Welcome to Cloudera Community. As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

    Translate
    Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
      Hasib1971
      Hasib1971
      Silver ⭐️⭐️
      February 14, 2026
      An invalid set of parameters has been specified in the url.
      Translate
      vafs
      Gold ⭐️⭐️⭐️
      vafs
      Gold ⭐️⭐️⭐️
      February 14, 2026

      Hello @Hasib1971, 

      Are you still facing an issue or that was what solved your error? 
      If you still have an issue, please open a new thread so we can help you there. 

      Translate
      Regards, Andrés Fallas -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs-up button.
      Terms & ConditionsAccessibility statement