Hi, I'm trying to add a new node to a 3 node cluster and I'm having a few issues. After adding a new node through Ambari, the new DataNode doesn't seem to be working properly. Ambari is showing the DataNode component as up, but the alerts for the DataNode process and DataNode Web UI are both down. Both are getting a connection failed ("Connection failed: [Errno 111] Connection refused to <my-new-node> ..."), I first thought it could be a network issue but everything seems to be fine. Inside the log files /var/log/hadoop/hs_err_pid#.log I can see the following errors: # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f5210b18b7c, pid=21338, tid=139990736889664 # # JRE version: (8.0_77-b03) (build ) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.77-b03 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x0000000001644800): JavaThread "Unknown thread" [_thread_in_Java, id=21338, stack(0x00007ffcc$ siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007ffcc64e8620 Registers: RAX=0x00007f520032a810, RBX=0x00007f520027bca8, RCX=0x0000000000000008, RDX=0x00007f52216190d0 RSP=0x00007ffcc64ec620, RBP=0x00007ffcc64ec660, RSI=0x0000000000000004, RDI=0x0000000001644800 R8 =0x0000000000000000, R9 =0x0000000000000003, R10=0x0000000000000000, R11=0x0000000000000004 R12=0x0000000000000000, R13=0x00007f520027bc98, R14=0x00007ffcc64ec668, R15=0x0000000001644800 RIP=0x00007f5210b18b7c, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000006 TRAPNO=0x000000000000000e Top of Stack: (sp=0x00007ffcc64ec620) 0x00007ffcc64ec620: 00007ffcc64ec620 00007f520027bc98 0x00007ffcc64ec630: 00007ffcc64ec668 00007f5200325d68 0x00007ffcc64ec640: 0000000000000000 00007f520027bca8 0x00007ffcc64ec650: 0000000000000000 00007ffcc64ec670 0x00007ffcc64ec660: 00007ffcc64ec6d0 00007f5210b084e7 0x00007ffcc64ec670: 00007ffc00001fa0 0000000000000000 0x00007ffcc64ec680: 0000000001644800 00007ffcc64ec750 0x00007ffcc64ec690: 00007f520027bca8 00007ffcc64ec710 0x00007ffcc64ec6a0: 00007ffcc64ec710 00007ffcc64ec8e8 0x00007ffcc64ec6b0: 00007f520000000a 00007f520027bca8 0x00007ffcc64ec6c0: 00007f5210b18940 00007ffcc64ec858 0x00007ffcc64ec6d0: 00007ffcc64ec840 00007f5220cf4616 0x00007ffcc64ec6e0: 0000000000000000 0000000001644800 0x00007ffcc64ec6f0: 00007ffcc64ec850 00007ffcc64ec8e0 0x00007ffcc64ec700: 00007f5210b18940 000000000000000a 0x00007ffcc64ec710: 0000000001644800 0000000001645e60 0x00007ffcc64ec720: 00007f520027bca8 0000000000000000 0x00007ffcc64ec730: 0000000000000000 0000000000000000 0x00007ffcc64ec740: 0000000000000000 00007ffcc64ec8e0 0x00007ffcc64ec750: 0000000001644800 0000000001644140 0x00007ffcc64ec760: 0000000001644180 00000000016441a0 0x00007ffcc64ec770: 0000000001644268 00000000000000d8 0x00007ffcc64ec780: 00007ffcc64eca30 00007ffcc64ec820 0x00007ffcc64ec790: 00007f520027bca8 0000000001644800 0x00007ffcc64ec7a0: 00007f5200000007 0000000001644800 0x00007ffcc64ec7b0: 00007f520027bca8 0000000001644800 0x00007ffcc64ec7c0: 00000000016451b0 00007f520027d690 0x00007ffcc64ec7d0: 00000008016450f0 0000000001644800 0x00007ffcc64ec7e0: 00007f520027bca8 0000000001644800 0x00007ffcc64ec7a0: 00007f5200000007 0000000001644800 0x00007ffcc64ec7b0: 00007f520027bca8 0000000001644800 0x00007ffcc64ec7c0: 00000000016451b0 00007f520027d690 0x00007ffcc64ec7d0: 00000008016450f0 0000000001644800 0x00007ffcc64ec7e0: 00007f520027bca8 0000000001644800 0x00007ffcc64ec7f0: 0000000001644800 00007f5220f15377 0x00007ffcc64ec800: 00007f520027bca8 0000000001644800 0x00007ffcc64ec810: 00007ffcc64ec900 00007ffcc64ec850 Instructions: (pc=0x00007f5210b18b7c) 0x00007f5210b18b5c: e1 f8 03 00 00 0f 84 bc 01 00 00 89 84 24 00 f0 0x00007f5210b18b6c: ff ff 89 84 24 00 e0 ff ff 89 84 24 00 d0 ff ff 0x00007f5210b18b7c: 89 84 24 00 c0 ff ff 89 84 24 00 b0 ff ff 89 84 0x00007f5210b18b8c: 24 00 a0 ff ff 89 84 24 00 90 ff ff 89 84 24 00 Register to memory mapping: RAX=0x00007f520032a810 is pointing into metadata RBX={method} {0x00007f520027bca8} '<clinit>' '()V' in 'java/lang/Object' RCX=0x0000000000000008 is an unknown value RDX=0x00007f52216190d0: <offset 0xfb10d0> in /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/server/libjvm.so at 0x00007f5$RSP=0x00007ffcc64ec620 is pointing into the stack for thread: 0x0000000001644800 RBP=0x00007ffcc64ec660 is pointing into the stack for thread: 0x0000000001644800 RSI=0x0000000000000004 is an unknown value RDI=0x0000000001644800 is a thread R8 =0x0000000000000000 is an unknown value R9 =0x0000000000000003 is an unknown value R10=0x0000000000000000 is an unknown value R11=0x0000000000000004 is an unknown value R12=0x0000000000000000 is an unknown value R13=0x00007f520027bc98 is pointing into metadata R14=0x00007ffcc64ec668 is pointing into the stack for thread: 0x0000000001644800 R15=0x0000000001644800 is a thread Stack: [0x00007ffcc63f1000,0x00007ffcc64f1000], sp=0x00007ffcc64ec620, free space=1005k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) j java.lang.Object.<clinit>()V+0 v ~StubRoutines::call_stub V [libjvm.so+0x68c616] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056 V [libjvm.so+0x63e967] InstanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*)+0xd7 V [libjvm.so+0x64122c] InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x1ac V [libjvm.so+0x6415f1] InstanceKlass::initialize(Thread*)+0x41 V [libjvm.so+0x64146d] InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x3ed V [libjvm.so+0x6415f1] InstanceKlass::initialize(Thread*)+0x41 V [libjvm.so+0xa67764] Threads::create_vm(JavaVMInitArgs*, bool*)+0x414 V [libjvm.so+0x6cd7e4] JNI_CreateJavaVM+0x74 C [jsvc.exec+0x639e] --------------- P R O C E S S --------------- Java Threads: ( => current thread ) =>0x0000000001644800 JavaThread "Unknown thread" [_thread_in_Java, id=21338, stack(0x00007ffcc63f1000,0x00007ff$ Other Threads: 0x0000000001757000 VMThread [stack: 0x00007f51ffedd000,0x00007f51fffde000] [id=21344] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap: par new generation total 184320K, used 3276K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000) eden space 163840K, 2% used [0x00000000c0000000, 0x00000000c0333340, 0x00000000ca000000) from space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000) to space 20480K, 0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000) concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x00000001000$ Metaspace used 777K, capacity 4480K, committed 4480K, reserved 1056768K class space used 75K, capacity 384K, committed 384K, reserved 1048576K Card table byte_map: [0x00007f5210244000,0x00007f5210445000] byte_map_base: 0x00007f520fc44000 Marking Bits: (CMSBitMap*) 0x000000000169cf98 Bits: [0x00007f5202f80000, 0x00007f5203c60000) Mod Union Table: (CMSBitMap*) 0x000000000169d058 Bits: [0x00007f5222111000, 0x00007f5222144800) Polling page: 0x00007f5222258000 CodeCache: size=245760Kb used=354Kb max_used=354Kb free=245405Kb bounds [0x00007f5210b08000, 0x00007f5210d78000, 0x00007f521fb08000] total_blobs=70 nmethods=0 adapters=48 compilation: enabled Compilation events (0 events): No events GC Heap History (0 events): No events Deoptimization events (0 events): No events Internal exceptions (0 events): No events Events (10 events): Event: 0.032 loading class java/lang/Short done Event: 0.032 loading class java/lang/Integer Event: 0.032 loading class java/lang/Integer done Event: 0.032 loading class java/lang/Long Event: 0.032 loading class java/lang/Long done Event: 0.033 loading class java/lang/NullPointerException Event: 0.033 loading class java/lang/NullPointerException done Event: 0.033 loading class java/lang/ArithmeticException Event: 0.033 loading class java/lang/ArithmeticException done Event: 0.035 Thread 0x0000000001644800 Thread added: 0x0000000001644800 Dynamic libraries: 00400000-0040b000 r-xp 00000000 fd:01 180355155 /usr/lib/bigtop-utils/jsvc 0060a000-0060b000 rw-p 0000a000 fd:01 180355155 /usr/lib/bigtop-utils/jsvc 0161f000-01640000 rw-p 00000000 00:00 0 [heap] 01640000-01777000 rw-p 00000000 00:00 0 [heap] c0000000-100060000 rw-p 00000000 00:00 0 100060000-140000000 ---p 00000000 00:00 0 7f51ffc58000-7f51ffedd000 rw-p 00000000 00:00 0 7f51ffedd000-7f51ffede000 ---p 00000000 00:00 0 7f51ffede000-7f520002d000 rw-p 00000000 00:00 0 [stack:21344] 7f520002d000-7f5200205000 r--s 03d1e000 fd:01 260051774 /usr/jdk64/jdk1.8.0_77/jre/lib/rt.jar 7f5200205000-7f520067b000 rw-p 00000000 00:00 0 7f520067b000-7f5200a7b000 ---p 00000000 00:00 0 7f5200a7b000-7f5200a7c000 ---p 00000000 00:00 0 7f5200a7c000-7f5204000000 rw-p 00000000 00:00 0 [stack:21343] 7f5204000000-7f5204021000 rw-p 00000000 00:00 0 7f5204021000-7f5208000000 ---p 00000000 00:00 0 7f5208000000-7f5208021000 rw-p 00000000 00:00 0 7f5208021000-7f520c000000 ---p 00000000 00:00 0 7f520c000000-7f520c021000 rw-p 00000000 00:00 0 7f520c021000-7f5210000000 ---p 00000000 00:00 0 7f5210042000-7f5210444000 rw-p 00000000 00:00 0 7f5210444000-7f5210445000 rw-p 00000000 00:00 0 7f5210445000-7f5210446000 ---p 00000000 00:00 0 7f5210446000-7f5210546000 rw-p 00000000 00:00 0 [stack:21342] 7f5210546000-7f5210547000 ---p 00000000 00:00 0 7f5210547000-7f5210647000 rw-p 00000000 00:00 0 [stack:21341] 7f5210647000-7f5210648000 ---p 00000000 00:00 0 7f5210648000-7f5210752000 rw-p 00000000 00:00 0 [stack:21340] 7f5210752000-7f5210b08000 ---p 00000000 00:00 0 7f5210b08000-7f5210d78000 rwxp 00000000 00:00 0 7f5210d78000-7f521fb08000 ---p 00000000 00:00 0 7f521fb08000-7f521fb22000 r-xp 00000000 fd:01 255885000 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521fb22000-7f521fd22000 ---p 0001a000 fd:01 255885000 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521fd22000-7f521fd23000 rw-p 0001a000 fd:01 255885000 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521fd23000-7f521fd4d000 r-xp 00000000 fd:01 255884995 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521fd4d000-7f521ff4d000 ---p 0002a000 fd:01 255884995 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521ff4d000-7f521ff4f000 rw-p 0002a000 fd:01 255884995 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521ff4f000-7f521ff5c000 r-xp 00000000 fd:01 255884999 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f521ff5c000-7f522015c000 ---p 0000d000 fd:01 255884999 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f522015c000-7f522015e000 rw-p 0000d000 fd:01 255884999 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/l$7f522015e000-7f5220165000 r-xp 00000000 fd:01 4388830 /usr/lib64/librt-2.17.so 7f5220165000-7f5220364000 ---p 00007000 fd:01 4388830 /usr/lib64/librt-2.17.so 7f5220364000-7f5220365000 r--p 00006000 fd:01 4388830 /usr/lib64/librt-2.17.so 7f5220365000-7f5220366000 rw-p 00007000 fd:01 4388830 /usr/lib64/librt-2.17.so 7f5220466000-7f5220666000 ---p 00100000 fd:01 4388808 /usr/lib64/libm-2.17.so 7f5220666000-7f5220667000 r--p 00100000 fd:01 4388808 /usr/lib64/libm-2.17.so 7f5220667000-7f5220668000 rw-p 00101000 fd:01 4388808 /usr/lib64/libm-2.17.so 7f5220668000-7f5221327000 r-xp 00000000 fd:01 264242951 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/s$7f5221327000-7f5221526000 ---p 00cbf000 fd:01 264242951 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/s$7f5221526000-7f52215ff000 rw-p 00cbe000 fd:01 264242951 /usr/jdk64/jdk1.8.0_77/jre/lib/amd64/s$7f52215ff000-7f5221648000 rw-p 00000000 00:00 0 7f5221648000-7f5221654000 r-xp 00000000 fd:01 4388818 /usr/lib64/libnss_files-2.17.so 7f5221654000-7f5221853000 ---p 0000c000 fd:01 4388818 /usr/lib64/libnss_files-2.17.so 7f5221853000-7f5221854000 r--p 0000b000 fd:01 4388818 /usr/lib64/libnss_files-2.17.so 7f5221854000-7f5221855000 rw-p 0000c000 fd:01 4388818 /usr/lib64/libnss_files-2.17.so 7f5221855000-7f522185b000 rw-p 00000000 00:00 0 7f522185b000-7f5221a12000 r-xp 00000000 fd:01 4258374 /usr/lib64/libc-2.17.so 7f5221a12000-7f5221c11000 ---p 001b7000 fd:01 4258374 /usr/lib64/libc-2.17.so 7f5221c11000-7f5221c15000 r--p 001b6000 fd:01 4258374 /usr/lib64/libc-2.17.so 7f5221c15000-7f5221c17000 rw-p 001ba000 fd:01 4258374 /usr/lib64/libc-2.17.so 7f5221c17000-7f5221c1c000 rw-p 00000000 00:00 0 7f5221c1c000-7f5221c33000 r-xp 00000000 fd:01 4388826 /usr/lib64/libpthread-2.17.so 7f5221c33000-7f5221e32000 ---p 00017000 fd:01 4388826 /usr/lib64/libpthread-2.17.so 7f5221e32000-7f5221e33000 r--p 00016000 fd:01 4388826 /usr/lib64/libpthread-2.17.so 7f5221e33000-7f5221e34000 rw-p 00017000 fd:01 4388826 /usr/lib64/libpthread-2.17.so 7f5221e34000-7f5221e38000 rw-p 00000000 00:00 0 7f5221e38000-7f5221e3a000 r-xp 00000000 fd:01 4388806 /usr/lib64/libdl-2.17.so 7f5221e3a000-7f522203a000 ---p 00002000 fd:01 4388806 /usr/lib64/libdl-2.17.so 7f522203a000-7f522203b000 r--p 00002000 fd:01 4388806 /usr/lib64/libdl-2.17.so 7f522203b000-7f522203c000 rw-p 00003000 fd:01 4388806 /usr/lib64/libdl-2.17.so 7f522203c000-7f522205c000 r-xp 00000000 fd:01 4388802 /usr/lib64/ld-2.17.so 7f5222066000-7f5222145000 rw-p 00000000 00:00 0 7f5222145000-7f5222146000 ---p 00000000 00:00 0 7f5222146000-7f5222246000 rw-p 00000000 00:00 0 [stack:21339] 7f5222246000-7f522224e000 rw-s 00000000 fd:01 75506322 /tmp/hsperfdata_hdfs/21338 7f522224e000-7f5222252000 rw-p 00000000 00:00 0 7f5222255000-7f5222258000 rw-p 00000000 00:00 0 7f5222258000-7f5222259000 r--p 00000000 00:00 0 7f5222259000-7f522225b000 rw-p 00000000 00:00 0 7f522225b000-7f522225c000 r--p 0001f000 fd:01 4388802 /usr/lib64/ld-2.17.so 7f522225c000-7f522225d000 rw-p 00020000 fd:01 4388802 /usr/lib64/ld-2.17.so 7f522225d000-7f522225e000 rw-p 00000000 00:00 0 7ffcc63ce000-7ffcc63f1000 rw-p 00000000 00:00 0 7ffcc63f1000-7ffcc63f4000 ---p 00000000 00:00 0 7ffcc64f4000-7ffcc64f1000 rw-p 00000000 00:00 0 [stack] 7ffcc6593000-7ffcc6595000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] VM Arguments: jvm_args: -Dproc_datanode -Xmx1024m -Dhdp.version=2.5.5.0-157 -Djava.net.preferIPv4Stack=true -Dhdp.version= -D$java_command: <unknown> java_class_path (initial): /usr/hdp/current/hadoop-client/conf:/usr/hdp/2.5.5.0-157/hadoop/lib/activation-1.1.j$Launcher Type: generic Environment Variables: JAVA_HOME=/usr/jdk64/jdk1.8.0_77 CLASSPATH=/usr/hdp/current/hadoop-client/conf:/usr/hdp/2.5.5.0-157/hadoop/lib/*:/usr/hdp/2.5.5.0-157/hadoop/.//$PATH=/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari$LD_LIBRARY_PATH=::/usr/hdp/2.5.5.0-157/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.5.5.0-157/hadoop/lib/native:$ Signal Handlers: SIGSEGV: [libjvm.so+0xabb070], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGBUS: [libjvm.so+0xabb070], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGFPE: [libjvm.so+0x918480], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGPIPE: [libjvm.so+0x918480], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGXFSZ: [libjvm.so+0x918480], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGILL: [libjvm.so+0x918480], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGUSR1: [+0x2370], sa_mask[0]=00000000010000000000000000000000, sa_flags=SA_RESTART SIGUSR2: [libjvm.so+0x919ab0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO SIGHUP: [+0x2370], sa_mask[0]=10000000000000000000000000000000, sa_flags=SA_RESTART SIGINT: [+0x2370], sa_mask[0]=01000000000000000000000000000000, sa_flags=SA_RESTART SIGTERM: [+0x2370], sa_mask[0]=00000000000000100000000000000000, sa_flags=SA_RESTART SIGQUIT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none --------------- S Y S T E M --------------- OS:CentOS Linux release 7.3.1611 (Core) uname:Linux 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 libc:glibc 2.17 NPTL 2.17 rlimit: STACK 8192k, CORE 0k, NPROC 127899, NOFILE 128000, AS infinity load average:0.00 0.01 0.05 /proc/meminfo: MemTotal: 32781464 kB MemFree: 31854368 kB MemAvailable: 31883208 kB Buffers: 872 kB Cached: 486048 kB SwapCached: 0 kB Active: 406548 kB Inactive: 193456 kB Active(anon): 177876 kB Inactive(anon): 107532 kB Active(file): 228672 kB Inactive(file): 85924 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 108844 kB Mapped: 77072 kB Shmem: 172324 kB Slab: 100536 kB SReclaimable: 65332 kB SUnreclaim: 35204 kB KernelStack: 2640 kB PageTables: 4284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 16390732 kB Committed_AS: 1753628 kB VmallocTotal: 34359738367 kB VmallocUsed: 61432 kB VmallocChunk: 34359666428 kB HardwareCorrupted: 0 kB AnonHugePages: 10240 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 98172 kB DirectMap2M: 3047424 kB DirectMap1G: 32505856 kB CPU:total 8 (1 cores per cpu, 1 threads per core) family 6 model 60 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2$ /proc/cpuinfo: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell, no TSX) stepping : 1 microcode : 0x1 cpu MHz : 2294.686 cache size : 4096 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips : 4589.37 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell, no TSX) stepping : 1 microcode : 0x1 cpu MHz : 2294.686 cache size : 4096 KB physical id : 1 siblings : 1 core id : 0 cpu cores : 1 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips : 4589.37 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell, no TSX) stepping : 1 microcode : 0x1 cpu MHz : 2294.686 cache size : 4096 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips : 4589.37 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell, no TSX) stepping : 1 microcode : 0x1 cpu MHz : 2294.686 physical id : 3 siblings : 1 core id : 0 cpu cores : 1 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips : 4589.37 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 4 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell, no TSX) stepping : 1 microcode : 0x1 cpu MHz : 2294.686 cache size : 4096 KB physical id : 4 siblings : 1 core id : 0 cpu cores : 1 apicid : 4 initial apicid : 4 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips : 4589.37 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 5 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell, no TSX) stepping : 1 microcode : 0x1 cpu MHz : 2294.686 cache size : 4096 KB physical id : 5 siblings : 1 core id : 0 cpu cores : 1 apicid : 5 initial apicid : 5 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips : 4589.37 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 6 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell, no TSX) stepping : 1 microcode : 0x1 cpu MHz : 2294.686 cache size : 4096 KB physical id : 6 siblings : 1 core id : 0 cpu cores : 1 apicid : 6 initial apicid : 6 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips : 4589.37 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell, no TSX) stepping : 1 microcode : 0x1 cpu MHz : 2294.686 cache size : 4096 KB physical id : 7 siblings : 1 core id : 0 cpu cores : 1 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse $bogomips : 4589.37 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: Memory: 4k page, physical 32781464k(31854368k free), swap 0k(0k free) vm_info: Java HotSpot(TM) 64-Bit Server VM (25.77-b03) for linux-amd64 JRE (1.8.0_77-b03), built on Mar 20 2016$ time: Sun Jun 25 21:16:16 2017 elapsed time: 0 seconds (0d 0h 0m 0s) Any ideas what could be going on? ---- UPDATE: Finally found the cause. Please visit: https://issues.apache.org/jira/browse/DAEMON-363 For the data node issue in particular I had to update HADOOP_DATANODE_OPTS environment varialbe in hadoop-env to have "-Xss2m" (I'm considering reducing this to -Xss1280k as mentioned on the case" ... View more