Namenode crashes with SEGFAULT when using JniBasedUnixGroupsMapping

[#1]

SYMPTOM

Namenode crash might be observed when jni based unix group mappings is enabled. The crash will usually generate the "hs_err" log file which will have the stack as below:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fbc814dd2a0, pid=380582, tid=140448021370624
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libnss_uxauth.so.2+0x4e2a0]  sqlite3ExprCodeTarget+0xcc3
#
......

Stack: [0x00007fbc9a5c5000,0x00007fbc9a6c6000],  sp=0x00007fbc9a6c2860,  free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libnss_uxauth.so.2+0x4e2a0]  sqlite3ExprCodeTarget+0xcc3
C  [libnss_uxauth.so.2+0x4e8db]  evalConstExpr+0xf7
C  [libnss_uxauth.so.2+0x47ae2]  sqlite3WalkExpr+0x34
C  [libnss_uxauth.so.2+0x47bdd]  sqlite3WalkExprList+0x42
C  [libnss_uxauth.so.2+0x47b80]  sqlite3WalkExpr+0xd2
C  [libnss_uxauth.so.2+0x47b15]  sqlite3WalkExpr+0x67
C  [libnss_uxauth.so.2+0x4e980]  sqlite3ExprCodeConstants+0x5a
C  [libnss_uxauth.so.2+0x7cac1]  sqlite3WhereBegin+0x1c5
C  [libnss_uxauth.so.2+0x6ecc6]  sqlite3Select+0x858
C  [libnss_uxauth.so.2+0x7ea58]  yy_reduce+0x86f
C  [libnss_uxauth.so.2+0x80f7c]  sqlite3Parser+0xc8
C  [libnss_uxauth.so.2+0x81d0d]  sqlite3RunParser+0x28b
C  [libnss_uxauth.so.2+0x677d2]  sqlite3Prepare+0x206
C  [libnss_uxauth.so.2+0x67ab1]  sqlite3LockAndPrepare+0x84
C  [libnss_uxauth.so.2+0x67c53]  sqlite3_prepare_v2+0x4d
C  [libnss_uxauth.so.2+0xad31]  init_usergroups+0x182
C  [libnss_uxauth.so.2+0x914c]  uxauth_initgroups+0x69
C  [libnss_uxauth.so.2+0xc9a0]  _nss_uxauth_initgroups_dyn+0x88
C  [libc.so.6+0xa979f]  __tls_get_addr@@GLIBC_2.3+0xa979f
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroupsForUser(Ljava/lang/String;)[Ljava/lang/String;+0
j  org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroups(Ljava/lang/String;)Ljava/util/List;+6
j  org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(Ljava/lang/String;)Ljava/util/List;+5
j  org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Ljava/lang/String;)Ljava/util/List;+19
j  org.apache.hadoop.security.Groups$GroupCacheLoader.load(Ljava/lang/String;)Ljava/util/List;+2
j  org.apache.hadoop.security.Groups$GroupCacheLoader.load(Ljava/lang/Object;)Ljava/lang/Object;+5
j  com.google.common.cache.CacheLoader.reload(Ljava/lang/Object;Ljava/lang/Object;)Lcom/google/common/util/concurrent/ListenableFuture;+2

ROOT CAUSE:

We have couple of Apache JIRA's which are reported that track this issue . https://issues.apache.org/jira/browse/HADOOP-10442 https://issues.apache.org/jira/browse/HADOOP-10527

WORK AROUND:

As a workaround we can change the JNI based mappings to shell based mapping by changing hadoop.security.group.mapping property through Ambari under "Advanced core-site" or in core-site.xml on namenode server. A HDFS restart would be required for this change to take effect.

<property>
  <name>hadoop.security.group.mapping</name>
  <value>org.apache.hadoop.security.ShellBasedUnixGroupsMapping</value>
</property>