Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Expert Contributor

SYMPTOM

Namenode crash might be observed when jni based unix group mappings is enabled. The crash will usually generate the "hs_err" log file which will have the stack as below:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fbc814dd2a0, pid=380582, tid=140448021370624
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libnss_uxauth.so.2+0x4e2a0]  sqlite3ExprCodeTarget+0xcc3
#
......

Stack: [0x00007fbc9a5c5000,0x00007fbc9a6c6000],  sp=0x00007fbc9a6c2860,  free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libnss_uxauth.so.2+0x4e2a0]  sqlite3ExprCodeTarget+0xcc3
C  [libnss_uxauth.so.2+0x4e8db]  evalConstExpr+0xf7
C  [libnss_uxauth.so.2+0x47ae2]  sqlite3WalkExpr+0x34
C  [libnss_uxauth.so.2+0x47bdd]  sqlite3WalkExprList+0x42
C  [libnss_uxauth.so.2+0x47b80]  sqlite3WalkExpr+0xd2
C  [libnss_uxauth.so.2+0x47b15]  sqlite3WalkExpr+0x67
C  [libnss_uxauth.so.2+0x4e980]  sqlite3ExprCodeConstants+0x5a
C  [libnss_uxauth.so.2+0x7cac1]  sqlite3WhereBegin+0x1c5
C  [libnss_uxauth.so.2+0x6ecc6]  sqlite3Select+0x858
C  [libnss_uxauth.so.2+0x7ea58]  yy_reduce+0x86f
C  [libnss_uxauth.so.2+0x80f7c]  sqlite3Parser+0xc8
C  [libnss_uxauth.so.2+0x81d0d]  sqlite3RunParser+0x28b
C  [libnss_uxauth.so.2+0x677d2]  sqlite3Prepare+0x206
C  [libnss_uxauth.so.2+0x67ab1]  sqlite3LockAndPrepare+0x84
C  [libnss_uxauth.so.2+0x67c53]  sqlite3_prepare_v2+0x4d
C  [libnss_uxauth.so.2+0xad31]  init_usergroups+0x182
C  [libnss_uxauth.so.2+0x914c]  uxauth_initgroups+0x69
C  [libnss_uxauth.so.2+0xc9a0]  _nss_uxauth_initgroups_dyn+0x88
C  [libc.so.6+0xa979f]  __tls_get_addr@@GLIBC_2.3+0xa979f
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroupsForUser(Ljava/lang/String;)[Ljava/lang/String;+0
j  org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroups(Ljava/lang/String;)Ljava/util/List;+6
j  org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(Ljava/lang/String;)Ljava/util/List;+5
j  org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Ljava/lang/String;)Ljava/util/List;+19
j  org.apache.hadoop.security.Groups$GroupCacheLoader.load(Ljava/lang/String;)Ljava/util/List;+2
j  org.apache.hadoop.security.Groups$GroupCacheLoader.load(Ljava/lang/Object;)Ljava/lang/Object;+5
j  com.google.common.cache.CacheLoader.reload(Ljava/lang/Object;Ljava/lang/Object;)Lcom/google/common/util/concurrent/ListenableFuture;+2

ROOT CAUSE:

We have couple of Apache JIRA's which are reported that track this issue . https://issues.apache.org/jira/browse/HADOOP-10442 https://issues.apache.org/jira/browse/HADOOP-10527

WORK AROUND:

As a workaround we can change the JNI based mappings to shell based mapping by changing hadoop.security.group.mapping property through Ambari under "Advanced core-site" or in core-site.xml on namenode server. A HDFS restart would be required for this change to take effect.
<property>
  <name>hadoop.security.group.mapping</name>
  <value>org.apache.hadoop.security.ShellBasedUnixGroupsMapping</value>
</property>
527 Views
0 Kudos
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎01-01-2017 07:37 PM
Updated by:
 
Contributors
Top Kudoed Authors