Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Super Collaborator

SYMPTOM

Namenode crash might be observed when jni based unix group mappings is enabled. The crash will usually generate the "hs_err" log file which will have the stack as below:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fbc814dd2a0, pid=380582, tid=140448021370624
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libnss_uxauth.so.2+0x4e2a0]  sqlite3ExprCodeTarget+0xcc3
#
......

Stack: [0x00007fbc9a5c5000,0x00007fbc9a6c6000],  sp=0x00007fbc9a6c2860,  free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libnss_uxauth.so.2+0x4e2a0]  sqlite3ExprCodeTarget+0xcc3
C  [libnss_uxauth.so.2+0x4e8db]  evalConstExpr+0xf7
C  [libnss_uxauth.so.2+0x47ae2]  sqlite3WalkExpr+0x34
C  [libnss_uxauth.so.2+0x47bdd]  sqlite3WalkExprList+0x42
C  [libnss_uxauth.so.2+0x47b80]  sqlite3WalkExpr+0xd2
C  [libnss_uxauth.so.2+0x47b15]  sqlite3WalkExpr+0x67
C  [libnss_uxauth.so.2+0x4e980]  sqlite3ExprCodeConstants+0x5a
C  [libnss_uxauth.so.2+0x7cac1]  sqlite3WhereBegin+0x1c5
C  [libnss_uxauth.so.2+0x6ecc6]  sqlite3Select+0x858
C  [libnss_uxauth.so.2+0x7ea58]  yy_reduce+0x86f
C  [libnss_uxauth.so.2+0x80f7c]  sqlite3Parser+0xc8
C  [libnss_uxauth.so.2+0x81d0d]  sqlite3RunParser+0x28b
C  [libnss_uxauth.so.2+0x677d2]  sqlite3Prepare+0x206
C  [libnss_uxauth.so.2+0x67ab1]  sqlite3LockAndPrepare+0x84
C  [libnss_uxauth.so.2+0x67c53]  sqlite3_prepare_v2+0x4d
C  [libnss_uxauth.so.2+0xad31]  init_usergroups+0x182
C  [libnss_uxauth.so.2+0x914c]  uxauth_initgroups+0x69
C  [libnss_uxauth.so.2+0xc9a0]  _nss_uxauth_initgroups_dyn+0x88
C  [libc.so.6+0xa979f]  __tls_get_addr@@GLIBC_2.3+0xa979f
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroupsForUser(Ljava/lang/String;)[Ljava/lang/String;+0
j  org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroups(Ljava/lang/String;)Ljava/util/List;+6
j  org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(Ljava/lang/String;)Ljava/util/List;+5
j  org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Ljava/lang/String;)Ljava/util/List;+19
j  org.apache.hadoop.security.Groups$GroupCacheLoader.load(Ljava/lang/String;)Ljava/util/List;+2
j  org.apache.hadoop.security.Groups$GroupCacheLoader.load(Ljava/lang/Object;)Ljava/lang/Object;+5
j  com.google.common.cache.CacheLoader.reload(Ljava/lang/Object;Ljava/lang/Object;)Lcom/google/common/util/concurrent/ListenableFuture;+2

ROOT CAUSE:

We have couple of Apache JIRA's which are reported that track this issue . https://issues.apache.org/jira/browse/HADOOP-10442 https://issues.apache.org/jira/browse/HADOOP-10527

WORK AROUND:

As a workaround we can change the JNI based mappings to shell based mapping by changing hadoop.security.group.mapping property through Ambari under "Advanced core-site" or in core-site.xml on namenode server. A HDFS restart would be required for this change to take effect.
<property>
  <name>hadoop.security.group.mapping</name>
  <value>org.apache.hadoop.security.ShellBasedUnixGroupsMapping</value>
</property>
1,311 Views
0 Kudos