Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problem starting a nodemanager

SOLVED Go to solution

Problem starting a nodemanager

New Contributor

Hi Folks,

 

I have a problem starting a node manager (actually two) the third is starting without any problem.

 

The failing node manager has this fatal error :

Error starting NodeManager
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-1006449310407885041.8: /tmp/libleveldbjni-64-1-1006449310407885041.8: failed to map segment from shared object: Operation not permitted]
at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:864)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:195)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:155)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:193)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)

 

We're using parcels and tried to verify permissions but dont see any problem. Even the process run from the agent is detecting the right version and path for the parcel. The host inspector is not reporting anuy problem.

 

Any help will be more than appreciated.

 

Best regards,

Mehdi

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Problem starting a nodemanager

Please verify that /tmp on both hosts do not have the noexec option set. You can verify this by running /bin/mount and checking the mount options. If you are able to, remount /tmp without noexec and try starting the NodeManager again
Regards,
Gautam Gopalakrishnan
7 REPLIES 7

Re: Problem starting a nodemanager

Please verify that /tmp on both hosts do not have the noexec option set. You can verify this by running /bin/mount and checking the mount options. If you are able to, remount /tmp without noexec and try starting the NodeManager again
Regards,
Gautam Gopalakrishnan

Re: Problem starting a nodemanager

New Contributor

Thank you very much Gautam for your quick answer.

 

Indeed I can see the noexec option set on the /tmp on the nodes having the problem.

 

We're fixing that and will let you know of what will come after removing the option.

 

Re: Problem starting a nodemanager

New Contributor

Now I can confirm that removing noexec has solved the problem.

By the way we removed it by executing as root mount -o remount,exec /tmp

 

Than you again Gautam

Re: Problem starting a nodemanager

New Contributor

Thanks for this reply, super helpful!

 

I just starting getting this same issue on only one of many NodeManager hosts. All the other node managers (like 60) have the noexec option set on the /tmp mount and worked fine, but one in particular didn't. Why does this sometimes fix it but sometimes things work fine? What interaction does the noexec option have with java native library loading?

Re: Problem starting a nodemanager

Explorer

Hi Gautam,

 

Is there any other way to fix this? I can not provide exec permission to /tmp in my environment due to some security reasons.

 

Helps much appriciated :)

Re: Problem starting a nodemanager

Vik11, you can work around this by setting Java's tmp dir to something other than /tmp. This solution has worked in the past for a customer, YMMV. Of course that mount point should not have noexec set.

 

In YARN configuration append '-Djava.io.tmpdir=/path/to/other/temp/dir' to the following properties:
1. ApplicationMaster Java Opts Base
2. Java Configuration Options for JobHistory Server
3. Java Configuration Options for NodeManager
4. Java Configuration Options for ResourceManager

For jobs:
Cloudera Manager --> YARN --> search for: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh and add this:
HADOOP_CLIENT_OPTS="-Djava.io.tmpdir=/path/to/other/temp/dir"

 

Now redeploy YARN client configuration.

Regards,
Gautam Gopalakrishnan

Re: Problem starting a nodemanager

Explorer

Great . Thanks Goutam :) 

 

So apart from yarn will there be any other CDH services (Hive, Pig, Flume, Spark, Impala, Hue, Oozie, Hbase) that too will require exec permission on /tmp ?

 

Having noexec on /tmp will have any problem to cluster functioning?

What would be your reccomandation here.

 

Thanks,