question Re: Problem starting a nodemanager in Support Questions

Problem starting a nodemanager

MehdiAbbes — Fri, 16 Sep 2022 09:29:34 GMT

Hi Folks,

I have a problem starting a node manager (actually two) the third is starting without any problem.

The failing node manager has this fatal error :

Error starting NodeManager
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-1006449310407885041.8: /tmp/libleveldbjni-64-1-1006449310407885041.8: failed to map segment from shared object: Operation not permitted]
at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:864)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:195)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:155)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:193)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)

We're using parcels and tried to verify permissions but dont see any problem. Even the process run from the agent is detecting the right version and path for the parcel. The host inspector is not reporting anuy problem.

Any help will be more than appreciated.

Best regards,

Mehdi

Re: Problem starting a nodemanager

GautamG — Tue, 19 May 2015 23:49:50 GMT

Please verify that /tmp on both hosts do not have the noexec option set. You can verify this by running /bin/mount and checking the mount options. If you are able to, remount /tmp without noexec and try starting the NodeManager again

Re: Problem starting a nodemanager

MehdiAbbes — Wed, 20 May 2015 09:11:26 GMT

Thank you very much Gautam for your quick answer.

Indeed I can see the noexec option set on the /tmp on the nodes having the problem.

We're fixing that and will let you know of what will come after removing the option.

Re: Problem starting a nodemanager

MehdiAbbes — Wed, 20 May 2015 10:22:10 GMT

Now I can confirm that removing noexec has solved the problem.

By the way we removed it by executing as root mount -o remount,exec /tmp

Than you again Gautam

Re: Problem starting a nodemanager

airhorns — Thu, 10 Dec 2015 21:33:57 GMT

Thanks for this reply, super helpful!

I just starting getting this same issue on only one of many NodeManager hosts. All the other node managers (like 60) have the noexec option set on the /tmp mount and worked fine, but one in particular didn't. Why does this sometimes fix it but sometimes things work fine? What interaction does the noexec option have with java native library loading?

Re: Problem starting a nodemanager

vik11 — Mon, 09 May 2016 17:22:27 GMT

Hi Gautam,

Is there any other way to fix this? I can not provide exec permission to /tmp in my environment due to some security reasons.

Helps much appriciated 🙂

Re: Problem starting a nodemanager

GautamG — Tue, 10 May 2016 07:11:37 GMT

Vik11, you can work around this by setting Java's tmp dir to something other than /tmp. This solution has worked in the past for a customer, YMMV. Of course that mount point should not have noexec set.

In YARN configuration append '-Djava.io.tmpdir=/path/to/other/temp/dir' to the following properties:
1. ApplicationMaster Java Opts Base
2. Java Configuration Options for JobHistory Server
3. Java Configuration Options for NodeManager
4. Java Configuration Options for ResourceManager

For jobs:
Cloudera Manager --> YARN --> search for: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh and add this:
HADOOP_CLIENT_OPTS="-Djava.io.tmpdir=/path/to/other/temp/dir"

Now redeploy YARN client configuration.

Re: Problem starting a nodemanager

vik11 — Tue, 10 May 2016 09:07:12 GMT

Great . Thanks Goutam 🙂

So apart from yarn will there be any other CDH services (Hive, Pig, Flume, Spark, Impala, Hue, Oozie, Hbase) that too will require exec permission on /tmp ?

Having noexec on /tmp will have any problem to cluster functioning?

What would be your reccomandation here.

Thanks,