HBase shell error: java.lang.UnsatisfiedLinkError

[#1]

On HDP 2.3.1, when trying to start hbase shell, got error message:

$ hbase shell java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: /tmp/jffi7007828132073784430.tmp: /tmp/jffi7007828132073784430.t mp: failed to map segment from shared object: Operation not permitted at com.kenai.jffi.Foreign$InValidInstanceHolder.getForeign(Foreign.java:90) at com.kenai.jffi.Foreign.getInstance(Foreign.java:95) at com.kenai.jffi.Library.openLibrary(Library.java:151) at com.kenai.jffi.Library.getCachedInstance(Library.java:125) at com.kenai.jaffl.provider.jffi.Library.loadNativeLibraries(Library.java:66) at com.kenai.jaffl.provider.jffi.Library.getNativeLibraries(Library.java:56) at com.kenai.jaffl.provider.jffi.Library.getSymbolAddress(Library.java:35) at com.kenai.jaffl.provider.jffi.Library.findSymbolAddress(Library.java:45) at com.kenai.jaffl.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:188) at com.kenai.jaffl.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:110) at com.kenai.jaffl.provider.jffi.Provider.loadLibrary(Provider.java:31) at com.kenai.jaffl.provider.jffi.Provider.loadLibrary(Provider.java:25) at com.kenai.jaffl.Library.loadLibrary(Library.java:76) at org.jruby.ext.posix.POSIXFactory$LinuxLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:108) at org.jruby.ext.posix.POSIXFactory$LinuxLibCProvider.getLibC(POSIXFactory.java:112) at org.jruby.ext.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:30) at org.jruby.ext.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:17) at org.jruby.ext.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:70) at org.jruby.ext.posix.POSIXFactory.loadPOSIX(POSIXFactory.java:31) at org.jruby.ext.posix.LazyPOSIX.loadPOSIX(LazyPOSIX.java:29) at org.jruby.ext.posix.LazyPOSIX.posix(LazyPOSIX.java:25) at org.jruby.ext.posix.LazyPOSIX.isatty(LazyPOSIX.java:159) at org.jruby.RubyIO.tty_p(RubyIO.java:1897) at org.jruby.RubyIO$i$0$0$tty_p.call(RubyIO$i$0$0$tty_p.gen:65535) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135) at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:63) at org.jruby.ast.IfNode.interpret(IfNode.java:111) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:147) at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:183) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135) at org.jruby.ast.VCallNode.interpret(VCallNode.java:86) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169) at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302) at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:148) at org.jruby.RubyClass.newInstance(RubyClass.java:822) at org.jruby.RubyClass$i$newInstance.call(RubyClass$i$newInstance.gen:65535) at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrNBlock.call(JavaMethod.java:249) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135) at usr.hdp.$2_dot_3_dot_2_dot_0_minus_2950.hbase.bin.hirb.__file__(/usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:128) at usr.hdp.$2_dot_3_dot_2_dot_0_minus_2950.hbase.bin.hirb.load(/usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb) at org.jruby.Ruby.runScript(Ruby.java:697) at org.jruby.Ruby.runScript(Ruby.java:690) at org.jruby.Ruby.runNormally(Ruby.java:597) at org.jruby.Ruby.runFromMain(Ruby.java:446) at org.jruby.Main.doRunFromMain(Main.java:369) at org.jruby.Main.internalRun(Main.java:258) at org.jruby.Main.run(Main.java:224) at org.jruby.Main.run(Main.java:208) at org.jruby.Main.main(Main.java:188) Caused by: java.lang.UnsatisfiedLinkError: /tmp/jffi7007828132073784430.tmp: /tmp/jffi7007828132073784430.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1086) at com.kenai.jffi.Init.loadFromJar(Init.java:164) at com.kenai.jffi.Init.load(Init.java:78) at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49) at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45) at com.kenai.jffi.Foreign.getInstance(Foreign.java:95) at com.kenai.jffi.Internals.getErrnoSaveFunction(Internals.java:44) at com.kenai.jaffl.provider.jffi.StubCompiler.getErrnoSaveFunction(StubCompiler.java:68) at com.kenai.jaffl.provider.jffi.StubCompiler.<clinit>(StubCompiler.java:18) at com.kenai.jaffl.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:146) ... 50 more SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBind er.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBi nder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Foreign.java:90:in `getForeign': java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: /tmp/jffi7007828132073784430.tm p: /tmp/jffi7007828132073784430.tmp: failed to map segment from shared object: Operation not permitted from Foreign.java:95:in `getInstance' from Library.java:151:in `openLibrary' from Library.java:125:in `getCachedInstance' from Library.java:66:in `loadNativeLibraries' from Library.java:56:in `getNativeLibraries' from Library.java:35:in `getSymbolAddress' from Library.java:45:in `findSymbolAddress' from DefaultInvokerFactory.java:51:in `createInvoker' from Library.java:27:in `getInvoker' from NativeInvocationHandler.java:90:in `createInvoker' from NativeInvocationHandler.java:74:in `getInvoker' from NativeInvocationHandler.java:110:in `invoke' from null:-1:in `isatty' from BaseNativePOSIX.java:300:in `isatty' from LazyPOSIX.java:159:in `isatty' from RubyIO.java:1897:in `tty_p' from RubyIO$i$0$0$tty_p.gen:65535:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from CallNoArgNode.java:63:in `interpret' from IfNode.java:111:in `interpret' from NewlineNode.java:104:in `interpret' from ASTInterpreter.java:74:in `INTERPRET_METHOD' from InterpretedMethod.java:147:in `call' from DefaultMethod.java:183:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from VCallNode.java:86:in `interpret' from NewlineNode.java:104:in `interpret' from BlockNode.java:71:in `interpret' from ASTInterpreter.java:74:in `INTERPRET_METHOD' from InterpretedMethod.java:169:in `call' from DefaultMethod.java:191:in `call' from CachingCallSite.java:302:in `cacheAndCall' from CachingCallSite.java:144:in `callBlock' from CachingCallSite.java:148:in `call' from RubyClass.java:822:in `newInstance' from RubyClass$i$newInstance.gen:65535:in `call' from JavaMethod.java:249:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:128:in `__file__' from /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:-1:in `load' from Ruby.java:697:in `runScript' from Ruby.java:690:in `runScript' from Ruby.java:597:in `runNormally' from Ruby.java:446:in `runFromMain' from Main.java:369:in `doRunFromMain' from Main.java:258:in `internalRun' from Main.java:224:in `run' from Main.java:208:in `run' from Main.java:188:in `main' Caused by: ClassLoader.java:-2:in `load': java.lang.UnsatisfiedLinkError: /tmp/jffi7007828132073784430.tmp: /tmp/jffi700782813207378443 0.tmp: failed to map segment from shared object: Operation not permitted from ClassLoader.java:1937:in `loadLibrary0' from ClassLoader.java:1822:in `loadLibrary' from Runtime.java:809:in `load0' from System.java:1086:in `load' from Init.java:164:in `loadFromJar' from Init.java:78:in `load' from Foreign.java:49:in `getInstanceHolder' from Foreign.java:45:in `<clinit>' from Foreign.java:95:in `getInstance' from Internals.java:44:in `getErrnoSaveFunction' from StubCompiler.java:68:in `getErrnoSaveFunction' from StubCompiler.java:18:in `<clinit>' from AsmLibraryLoader.java:146:in `generateInterfaceImpl' from AsmLibraryLoader.java:110:in `loadLibrary' from Provider.java:31:in `loadLibrary' from Provider.java:25:in `loadLibrary' from Library.java:76:in `loadLibrary' from POSIXFactory.java:108:in `<clinit>' from POSIXFactory.java:112:in `getLibC' from BaseNativePOSIX.java:30:in `<init>' from LinuxPOSIX.java:17:in `<init>' from POSIXFactory.java:70:in `loadLinuxPOSIX' from POSIXFactory.java:31:in `loadPOSIX' from LazyPOSIX.java:29:in `loadPOSIX' from LazyPOSIX.java:25:in `posix' from LazyPOSIX.java:159:in `isatty' from RubyIO.java:1897:in `tty_p' from RubyIO$i$0$0$tty_p.gen:65535:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from CallNoArgNode.java:63:in `interpret' from IfNode.java:111:in `interpret' from NewlineNode.java:104:in `interpret' from ASTInterpreter.java:74:in `INTERPRET_METHOD' from InterpretedMethod.java:147:in `call' from DefaultMethod.java:183:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from VCallNode.java:86:in `interpret' from NewlineNode.java:104:in `interpret' from BlockNode.java:71:in `interpret' from ASTInterpreter.java:74:in `INTERPRET_METHOD' from InterpretedMethod.java:169:in `call' from DefaultMethod.java:191:in `call' from CachingCallSite.java:302:in `cacheAndCall' from CachingCallSite.java:144:in `callBlock' from CachingCallSite.java:148:in `call' from RubyClass.java:822:in `newInstance' from RubyClass$i$newInstance.gen:65535:in `call' from JavaMethod.java:249:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:128:in `__file__' from /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:-1:in `load' from Ruby.java:697:in `runScript' from Ruby.java:690:in `runScript' from Ruby.java:597:in `runNormally' from Ruby.java:446:in `runFromMain' from Main.java:369:in `doRunFromMain' from Main.java:258:in `internalRun' from Main.java:224:in `run' from Main.java:208:in `run' from Main.java:188:in `main'

[#2 · Accepted solution]

Hello Mike,

Check that /tmp is not mounted with the noexec flag on that node.

sudo mount | grep /tmp

If so, remounting without that option should fix this.

If removing noexec isn't an option, you can control the directory Java uses for temporary storage through the java.io.tmpdir system property. Give the following a try, replacing the directory with your home directory or another filesystem without the noexec flag.

hbase -Djava.io.tmpdir=/some/other/writable/directory shell

[#3]

Thank you Shane. Find out our company security policy block execution on tmp directory. So We created a directory for hbase.tmp.dir and points it to that one in Ambari setting, it worked.

Thanks,