HDP3.0 hbase fail to start due to noexec /tmp

[#1]

Hi,I installed hadoop cluster using HDP3.0 but hbase does not work due to this error:

Caused by: java.lang.UnsatisfiedLinkError: failed to load the required native library
at org.apache.hbase.thirdparty.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:81)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:55)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
at org.apache.hbase.thirdparty.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68)
at org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.<init>(NettyEventLoopGroupConfig.java:61)
at org.apache.hadoop.hbase.regionserver.HRegionServer.setupNetty(HRegionServer.java:673)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:532)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:472)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2923)
... 5 more
Caused by: java.lang.UnsatisfiedLinkError: /tmp/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_644257058602762792223.so: /tmp/liborg_apache_hbase_thirdparty_netty_transport_native_ep
oll_x86_644257058602762792223.so: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:187)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:207)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.<clinit>(Native.java:65)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:33)
... 24 more

Looks it is caused by nonexec permission of /tmp. To make hbase not use /tmp, I added:

{ "hbase-site": { "properties": { "hbase.tmp.dir": "/u01/tmp" } } }

But this does not work and hbase still uses /tmp. Any idea? Appreciate any help.

[#2]

Hi @Lian Jiang ,

Can you give some more details about the issue.

Like your operating system, Internal memory

are you trying to deploy the HDP3.0 using cloudbreak ?

Additionally, this error is at Netty, a third party app used by HDP services to deploy containers .

I found something similar here : https://github.com/netty/netty/issues/6678

Seems this is related.

[#3]

Thanks Akhil S Naik

OS: oracle linux 7,

memory: 314g

deployment tool: ambari 2.7 blueprint.

Is HDP3.0 using the netty having bug 6678 fixed? 6678 seems to be just a warning. But it failed hbase in my case. Any solution? Thanks.

[#4 · Accepted solution]

This problem is resolved by adding hbase.netty.nativetransport = false.

Do you foresee any issue using this fix? @Akhil S Naik

Thanks.

[UPDATE 8/15/2018]

This setting works and may be better than the above one:

{ "hbase-env": { "properties": { "hbase_java_io_tmpdir": "/u01/tmp" } } },

[#5]

Hi @Lian Jiang,

Glad that your issue resolved.

changing the hbase temp directory solution is best IMHO .

[#6]

@Lian Jiang

For other HCC users reference:

Adding a link to the other thread which describes about this issue in more detail.

https://community.hortonworks.com/questions/212329/hdp30-timeline-service-v2-reader-cannot-create-zo...