Support Questions

alanfeat · ‎06-26-2017

Hi,

I'm trying to add a new node to a 3 node cluster and I'm having a few issues. After adding a new node through Ambari, the new DataNode doesn't seem to be working properly. Ambari is showing the DataNode component as up, but the alerts for the DataNode process and DataNode Web UI are both down. Both are getting a connection failed ("Connection failed: [Errno 111] Connection refused to <my-new-node> ..."), I first thought it could be a network issue but everything seems to be fine.

Inside the log files /var/log/hadoop/hs_err_pid#.log I can see the following errors:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f5210b18b7c, pid=21338, tid=139990736889664
#
# JRE version:  (8.0_77-b03) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.77-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# j  java.lang.Object.<clinit>()V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------


Current thread (0x0000000001644800):  JavaThread "Unknown thread" [_thread_in_Java, id=21338, stack(0x00007ffcc$


siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007ffcc64e8620


Registers:
RAX=0x00007f520032a810, RBX=0x00007f520027bca8, RCX=0x0000000000000008, RDX=0x00007f52216190d0
RSP=0x00007ffcc64ec620, RBP=0x00007ffcc64ec660, RSI=0x0000000000000004, RDI=0x0000000001644800
R8 =0x0000000000000000, R9 =0x0000000000000003, R10=0x0000000000000000, R11=0x0000000000000004
R12=0x0000000000000000, R13=0x00007f520027bc98, R14=0x00007ffcc64ec668, R15=0x0000000001644800
RIP=0x00007f5210b18b7c, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e


Top of Stack: (sp=0x00007ffcc64ec620)
0x00007ffcc64ec620:   00007ffcc64ec620 00007f520027bc98
0x00007ffcc64ec630:   00007ffcc64ec668 00007f5200325d68
0x00007ffcc64ec640:   0000000000000000 00007f520027bca8
0x00007ffcc64ec650:   0000000000000000 00007ffcc64ec670
0x00007ffcc64ec660:   00007ffcc64ec6d0 00007f5210b084e7
0x00007ffcc64ec670:   00007ffc00001fa0 0000000000000000
0x00007ffcc64ec680:   0000000001644800 00007ffcc64ec750
0x00007ffcc64ec690:   00007f520027bca8 00007ffcc64ec710
0x00007ffcc64ec6a0:   00007ffcc64ec710 00007ffcc64ec8e8
0x00007ffcc64ec6b0:   00007f520000000a 00007f520027bca8
0x00007ffcc64ec6c0:   00007f5210b18940 00007ffcc64ec858
0x00007ffcc64ec6d0:   00007ffcc64ec840 00007f5220cf4616
0x00007ffcc64ec6e0:   0000000000000000 0000000001644800
0x00007ffcc64ec6f0:   00007ffcc64ec850 00007ffcc64ec8e0
0x00007ffcc64ec700:   00007f5210b18940 000000000000000a
0x00007ffcc64ec710:   0000000001644800 0000000001645e60
0x00007ffcc64ec720:   00007f520027bca8 0000000000000000
0x00007ffcc64ec730:   0000000000000000 0000000000000000
0x00007ffcc64ec740:   0000000000000000 00007ffcc64ec8e0
0x00007ffcc64ec750:   0000000001644800 0000000001644140
0x00007ffcc64ec760:   0000000001644180 00000000016441a0
0x00007ffcc64ec770:   0000000001644268 00000000000000d8
0x00007ffcc64ec780:   00007ffcc64eca30 00007ffcc64ec820
0x00007ffcc64ec790:   00007f520027bca8 0000000001644800
0x00007ffcc64ec7a0:   00007f5200000007 0000000001644800
0x00007ffcc64ec7b0:   00007f520027bca8 0000000001644800
0x00007ffcc64ec7c0:   00000000016451b0 00007f520027d690
0x00007ffcc64ec7d0:   00000008016450f0 0000000001644800
0x00007ffcc64ec7e0:   00007f520027bca8 0000000001644800
0x00007ffcc64ec7a0:   00007f5200000007 0000000001644800
0x00007ffcc64ec7b0:   00007f520027bca8 0000000001644800
0x00007ffcc64ec7c0:   00000000016451b0 00007f520027d690
0x00007ffcc64ec7d0:   00000008016450f0 0000000001644800
0x00007ffcc64ec7e0:   00007f520027bca8 0000000001644800
0x00007ffcc64ec7f0:   0000000001644800 00007f5220f15377
0x00007ffcc64ec800:   00007f520027bca8 0000000001644800
0x00007ffcc64ec810:   00007ffcc64ec900 00007ffcc64ec850


Instructions: (pc=0x00007f5210b18b7c)
0x00007f5210b18b5c:   e1 f8 03 00 00 0f 84 bc 01 00 00 89 84 24 00 f0
0x00007f5210b18b6c:   ff ff 89 84 24 00 e0 ff ff 89 84 24 00 d0 ff ff
0x00007f5210b18b7c:   89 84 24 00 c0 ff ff 89 84 24 00 b0 ff ff 89 84
0x00007f5210b18b8c:   24 00 a0 ff ff 89 84 24 00 90 ff ff 89 84 24 00


Register to memory mapping:


RAX=0x00007f520032a810 is pointing into metadata
RBX={method} {0x00007f520027bca8} '<clinit>' '()V' in 'java/lang/Object'
RCX=0x0000000000000008 is an unknown value
RDX=0x00007f52216190d0: <offset 0xfb10d0> in /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/server/libjvm.so at 0x00007f5$RSP=0x00007ffcc64ec620 is pointing into the stack for thread: 0x0000000001644800
RBP=0x00007ffcc64ec660 is pointing into the stack for thread: 0x0000000001644800
RSI=0x0000000000000004 is an unknown value
RDI=0x0000000001644800 is a thread
R8 =0x0000000000000000 is an unknown value
R9 =0x0000000000000003 is an unknown value
R10=0x0000000000000000 is an unknown value
R11=0x0000000000000004 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x00007f520027bc98 is pointing into metadata
R14=0x00007ffcc64ec668 is pointing into the stack for thread: 0x0000000001644800
R15=0x0000000001644800 is a thread




Stack: [0x00007ffcc63f1000,0x00007ffcc64f1000],  sp=0x00007ffcc64ec620,  free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
j  java.lang.Object.<clinit>()V+0
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68c616]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x63e967]  InstanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*)+0xd7
V  [libjvm.so+0x64122c]  InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x1ac
V  [libjvm.so+0x6415f1]  InstanceKlass::initialize(Thread*)+0x41
V  [libjvm.so+0x64146d]  InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x3ed
V  [libjvm.so+0x6415f1]  InstanceKlass::initialize(Thread*)+0x41
V  [libjvm.so+0xa67764]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x414
V  [libjvm.so+0x6cd7e4]  JNI_CreateJavaVM+0x74
C  [jsvc.exec+0x639e]




---------------  P R O C E S S  ---------------


Java Threads: ( => current thread )
=>0x0000000001644800 JavaThread "Unknown thread" [_thread_in_Java, id=21338, stack(0x00007ffcc63f1000,0x00007ff$


Other Threads:
  0x0000000001757000 VMThread [stack: 0x00007f51ffedd000,0x00007f51fffde000] [id=21344]


VM state:not at safepoint (normal execution)


VM Mutex/Monitor currently owned by a thread: None


Heap:
 par new generation   total 184320K, used 3276K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,   2% used [0x00000000c0000000, 0x00000000c0333340, 0x00000000ca000000)
  from space 20480K,   0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x00000001000$ Metaspace       used 777K, capacity 4480K, committed 4480K, reserved 1056768K
  class space    used 75K, capacity 384K, committed 384K, reserved 1048576K


Card table byte_map: [0x00007f5210244000,0x00007f5210445000] byte_map_base: 0x00007f520fc44000


Marking Bits: (CMSBitMap*) 0x000000000169cf98
 Bits: [0x00007f5202f80000, 0x00007f5203c60000)


Mod Union Table: (CMSBitMap*) 0x000000000169d058
 Bits: [0x00007f5222111000, 0x00007f5222144800)


Polling page: 0x00007f5222258000


CodeCache: size=245760Kb used=354Kb max_used=354Kb free=245405Kb
 bounds [0x00007f5210b08000, 0x00007f5210d78000, 0x00007f521fb08000]
 total_blobs=70 nmethods=0 adapters=48
 compilation: enabled


Compilation events (0 events):
No events


GC Heap History (0 events):
No events


Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events


Events (10 events):
Event: 0.032 loading class java/lang/Short done
Event: 0.032 loading class java/lang/Integer
Event: 0.032 loading class java/lang/Integer done
Event: 0.032 loading class java/lang/Long
Event: 0.032 loading class java/lang/Long done
Event: 0.033 loading class java/lang/NullPointerException
Event: 0.033 loading class java/lang/NullPointerException done
Event: 0.033 loading class java/lang/ArithmeticException
Event: 0.033 loading class java/lang/ArithmeticException done
Event: 0.035 Thread 0x0000000001644800 Thread added: 0x0000000001644800




Dynamic libraries:
00400000-0040b000 r-xp 00000000 fd:01 180355155                          /usr/lib/bigtop-utils/jsvc
0060a000-0060b000 rw-p 0000a000 fd:01 180355155                          /usr/lib/bigtop-utils/jsvc
0161f000-01640000 rw-p 00000000 00:00 0                                  [heap]
01640000-01777000 rw-p 00000000 00:00 0                                  [heap]
c0000000-100060000 rw-p 00000000 00:00 0
100060000-140000000 ---p 00000000 00:00 0
7f51ffc58000-7f51ffedd000 rw-p 00000000 00:00 0
7f51ffedd000-7f51ffede000 ---p 00000000 00:00 0
7f51ffede000-7f520002d000 rw-p 00000000 00:00 0                          [stack:21344]
7f520002d000-7f5200205000 r--s 03d1e000 fd:01 260051774                  /usr/jdk64/jdk1.8.0_77/jre/lib/rt.jar
7f5200205000-7f520067b000 rw-p 00000000 00:00 0
7f520067b000-7f5200a7b000 ---p 00000000 00:00 0
7f5200a7b000-7f5200a7c000 ---p 00000000 00:00 0
7f5200a7c000-7f5204000000 rw-p 00000000 00:00 0                          [stack:21343]
7f5204000000-7f5204021000 rw-p 00000000 00:00 0
7f5204021000-7f5208000000 ---p 00000000 00:00 0
7f5208000000-7f5208021000 rw-p 00000000 00:00 0
7f5208021000-7f520c000000 ---p 00000000 00:00 0
7f520c000000-7f520c021000 rw-p 00000000 00:00 0
7f520c021000-7f5210000000 ---p 00000000 00:00 0
7f5210042000-7f5210444000 rw-p 00000000 00:00 0
7f5210444000-7f5210445000 rw-p 00000000 00:00 0
7f5210445000-7f5210446000 ---p 00000000 00:00 0
7f5210446000-7f5210546000 rw-p 00000000 00:00 0                          [stack:21342]
7f5210546000-7f5210547000 ---p 00000000 00:00 0
7f5210547000-7f5210647000 rw-p 00000000 00:00 0                          [stack:21341]
7f5210647000-7f5210648000 ---p 00000000 00:00 0
7f5210648000-7f5210752000 rw-p 00000000 00:00 0                          [stack:21340]
7f5210752000-7f5210b08000 ---p 00000000 00:00 0
7f5210b08000-7f5210d78000 rwxp 00000000 00:00 0
7f5210d78000-7f521fb08000 ---p 00000000 00:00 0
7f521fb08000-7f521fb22000 r-xp 00000000 fd:01 255885000                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521fb22000-7f521fd22000 ---p 0001a000 fd:01 255885000                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521fd22000-7f521fd23000 rw-p 0001a000 fd:01 255885000                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521fd23000-7f521fd4d000 r-xp 00000000 fd:01 255884995                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521fd4d000-7f521ff4d000 ---p 0002a000 fd:01 255884995                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521ff4d000-7f521ff4f000 rw-p 0002a000 fd:01 255884995                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521ff4f000-7f521ff5c000 r-xp 00000000 fd:01 255884999                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521ff5c000-7f522015c000 ---p 0000d000 fd:01 255884999                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f522015c000-7f522015e000 rw-p 0000d000 fd:01 255884999                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f522015e000-7f5220165000 r-xp 00000000 fd:01 4388830                    /usr/lib64/librt-2.17.so
7f5220165000-7f5220364000 ---p 00007000 fd:01 4388830                    /usr/lib64/librt-2.17.so
7f5220364000-7f5220365000 r--p 00006000 fd:01 4388830                    /usr/lib64/librt-2.17.so
7f5220365000-7f5220366000 rw-p 00007000 fd:01 4388830                    /usr/lib64/librt-2.17.so
7f5220466000-7f5220666000 ---p 00100000 fd:01 4388808                    /usr/lib64/libm-2.17.so
7f5220666000-7f5220667000 r--p 00100000 fd:01 4388808                    /usr/lib64/libm-2.17.so
7f5220667000-7f5220668000 rw-p 00101000 fd:01 4388808                    /usr/lib64/libm-2.17.so
7f5220668000-7f5221327000 r-xp 00000000 fd:01 264242951                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/s$7f5221327000-7f5221526000 ---p 00cbf000 fd:01 264242951                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/s$7f5221526000-7f52215ff000 rw-p 00cbe000 fd:01 264242951                  /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/s$7f52215ff000-7f5221648000 rw-p 00000000 00:00 0
7f5221648000-7f5221654000 r-xp 00000000 fd:01 4388818                    /usr/lib64/libnss_files-2.17.so
7f5221654000-7f5221853000 ---p 0000c000 fd:01 4388818                    /usr/lib64/libnss_files-2.17.so
7f5221853000-7f5221854000 r--p 0000b000 fd:01 4388818                    /usr/lib64/libnss_files-2.17.so
7f5221854000-7f5221855000 rw-p 0000c000 fd:01 4388818                    /usr/lib64/libnss_files-2.17.so
7f5221855000-7f522185b000 rw-p 00000000 00:00 0
7f522185b000-7f5221a12000 r-xp 00000000 fd:01 4258374                    /usr/lib64/libc-2.17.so
7f5221a12000-7f5221c11000 ---p 001b7000 fd:01 4258374                    /usr/lib64/libc-2.17.so
7f5221c11000-7f5221c15000 r--p 001b6000 fd:01 4258374                    /usr/lib64/libc-2.17.so
7f5221c15000-7f5221c17000 rw-p 001ba000 fd:01 4258374                    /usr/lib64/libc-2.17.so
7f5221c17000-7f5221c1c000 rw-p 00000000 00:00 0
7f5221c1c000-7f5221c33000 r-xp 00000000 fd:01 4388826                    /usr/lib64/libpthread-2.17.so
7f5221c33000-7f5221e32000 ---p 00017000 fd:01 4388826                    /usr/lib64/libpthread-2.17.so
7f5221e32000-7f5221e33000 r--p 00016000 fd:01 4388826                    /usr/lib64/libpthread-2.17.so
7f5221e33000-7f5221e34000 rw-p 00017000 fd:01 4388826                    /usr/lib64/libpthread-2.17.so
7f5221e34000-7f5221e38000 rw-p 00000000 00:00 0
7f5221e38000-7f5221e3a000 r-xp 00000000 fd:01 4388806                    /usr/lib64/libdl-2.17.so
7f5221e3a000-7f522203a000 ---p 00002000 fd:01 4388806                    /usr/lib64/libdl-2.17.so
7f522203a000-7f522203b000 r--p 00002000 fd:01 4388806                    /usr/lib64/libdl-2.17.so
7f522203b000-7f522203c000 rw-p 00003000 fd:01 4388806                    /usr/lib64/libdl-2.17.so
7f522203c000-7f522205c000 r-xp 00000000 fd:01 4388802                    /usr/lib64/ld-2.17.so
7f5222066000-7f5222145000 rw-p 00000000 00:00 0
7f5222145000-7f5222146000 ---p 00000000 00:00 0
7f5222146000-7f5222246000 rw-p 00000000 00:00 0                          [stack:21339]
7f5222246000-7f522224e000 rw-s 00000000 fd:01 75506322                   /tmp/hsperfdata_hdfs/21338
7f522224e000-7f5222252000 rw-p 00000000 00:00 0
7f5222255000-7f5222258000 rw-p 00000000 00:00 0
7f5222258000-7f5222259000 r--p 00000000 00:00 0
7f5222259000-7f522225b000 rw-p 00000000 00:00 0
7f522225b000-7f522225c000 r--p 0001f000 fd:01 4388802                    /usr/lib64/ld-2.17.so
7f522225c000-7f522225d000 rw-p 00020000 fd:01 4388802                    /usr/lib64/ld-2.17.so
7f522225d000-7f522225e000 rw-p 00000000 00:00 0
7ffcc63ce000-7ffcc63f1000 rw-p 00000000 00:00 0
7ffcc63f1000-7ffcc63f4000 ---p 00000000 00:00 0
7ffcc64f4000-7ffcc64f1000 rw-p 00000000 00:00 0                          [stack]
7ffcc6593000-7ffcc6595000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


VM Arguments:
jvm_args: -Dproc_datanode -Xmx1024m -Dhdp.version=2.5.5.0-157 -Djava.net.preferIPv4Stack=true -Dhdp.version= -D$java_command: <unknown>
java_class_path (initial): /usr/hdp/current/hadoop-client/conf:/usr/hdp/2.5.5.0-157/hadoop/lib/activation-1.1.j$Launcher Type: generic


Environment Variables:
JAVA_HOME=/usr/jdk64/jdk1.8.0_77
CLASSPATH=/usr/hdp/current/hadoop-client/conf:/usr/hdp/2.5.5.0-157/hadoop/lib/*:/usr/hdp/2.5.5.0-157/hadoop/.//$PATH=/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari$LD_LIBRARY_PATH=::/usr/hdp/2.5.5.0-157/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.5.5.0-157/hadoop/lib/native:$


Signal Handlers:
SIGSEGV: [libjvm.so+0xabb070], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xabb070], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x918480], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x918480], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x918480], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x918480], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: [+0x2370], sa_mask[0]=00000000010000000000000000000000, sa_flags=SA_RESTART
SIGUSR2: [libjvm.so+0x919ab0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [+0x2370], sa_mask[0]=10000000000000000000000000000000, sa_flags=SA_RESTART
SIGINT: [+0x2370], sa_mask[0]=01000000000000000000000000000000, sa_flags=SA_RESTART
SIGTERM: [+0x2370], sa_mask[0]=00000000000000100000000000000000, sa_flags=SA_RESTART
SIGQUIT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none




---------------  S Y S T E M  ---------------


OS:CentOS Linux release 7.3.1611 (Core)


uname:Linux 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 127899, NOFILE 128000, AS infinity
load average:0.00 0.01 0.05


/proc/meminfo:
MemTotal:       32781464 kB
MemFree:        31854368 kB
MemAvailable:   31883208 kB
Buffers:             872 kB
Cached:           486048 kB
SwapCached:            0 kB
Active:           406548 kB
Inactive:         193456 kB
Active(anon):     177876 kB
Inactive(anon):   107532 kB
Active(file):     228672 kB
Inactive(file):    85924 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        108844 kB
Mapped:            77072 kB
Shmem:            172324 kB
Slab:             100536 kB
SReclaimable:      65332 kB
SUnreclaim:        35204 kB
KernelStack:        2640 kB
PageTables:         4284 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16390732 kB
Committed_AS:    1753628 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       61432 kB
VmallocChunk:   34359666428 kB
HardwareCorrupted:     0 kB
AnonHugePages:     10240 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       98172 kB
DirectMap2M:     3047424 kB
DirectMap1G:    32505856 kB




CPU:total 8 (1 cores per cpu, 1 threads per core) family 6 model 60 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2$


/proc/cpuinfo:
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel Core Processor (Haswell, no TSX)
stepping        : 1
microcode       : 0x1
cpu MHz         : 2294.686
cache size      : 4096 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel Core Processor (Haswell, no TSX)
stepping        : 1
microcode       : 0x1
cpu MHz         : 2294.686
cache size      : 4096 KB
physical id     : 1
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel Core Processor (Haswell, no TSX)
stepping        : 1
microcode       : 0x1
cpu MHz         : 2294.686
cache size      : 4096 KB
physical id     : 2
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel Core Processor (Haswell, no TSX)
stepping        : 1
microcode       : 0x1
cpu MHz         : 2294.686
physical id     : 3
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


processor       : 4
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel Core Processor (Haswell, no TSX)
stepping        : 1
microcode       : 0x1
cpu MHz         : 2294.686
cache size      : 4096 KB
physical id     : 4
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


processor       : 5
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel Core Processor (Haswell, no TSX)
stepping        : 1
microcode       : 0x1
cpu MHz         : 2294.686
cache size      : 4096 KB
physical id     : 5
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


processor       : 6
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel Core Processor (Haswell, no TSX)
stepping        : 1
microcode       : 0x1
cpu MHz         : 2294.686
cache size      : 4096 KB
physical id     : 6
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel Core Processor (Haswell, no TSX)
stepping        : 1
microcode       : 0x1
cpu MHz         : 2294.686
cache size      : 4096 KB
physical id     : 7
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:






Memory: 4k page, physical 32781464k(31854368k free), swap 0k(0k free)


vm_info: Java HotSpot(TM) 64-Bit Server VM (25.77-b03) for linux-amd64 JRE (1.8.0_77-b03), built on Mar 20 2016$


time: Sun Jun 25 21:16:16 2017
elapsed time: 0 seconds (0d 0h 0m 0s)

Any ideas what could be going on?

----

UPDATE: Finally found the cause. Please visit: https://issues.apache.org/jira/browse/DAEMON-363

For the data node issue in particular I had to update HADOOP_DATANODE_OPTS environment varialbe in hadoop-env to have "-Xss2m" (I'm considering reducing this to -Xss1280k as mentioned on the case"

james_switzer · ‎06-26-2017

@Alan Featherston I was having the same issue after applying some OS/Kernel patches. Your article helped me to resolve it. Thank you for posting this!

View solution in original post

james_switzer · ‎06-26-2017

@Alan Featherston I was having the same issue after applying some OS/Kernel patches. Your article helped me to resolve it. Thank you for posting this!

demarkle · ‎06-27-2017

In our case it was related to this stack guard kernel patch:

CVE-2017-1000364 kernel: heap/stack gap jumping via unbounded stack allocations

https://access.redhat.com/errata/RHSA-2017:1484

stephen_coyle · ‎06-29-2017

im having the same issue. im going to give your solution a try. @Alan Featherston where do you change the hadoop_datanode_opts? im using ambari to control everything as well. thanks

dms1122 · ‎10-17-2017

Thank you, it has helped to resolve the issue.

Cloudera Community

Support Questions

DataNode failing to start - JRE SIGBUS error