recently, there are a few jobs failed randomly everyday because the map/reduce task connection timeout with app master.  the below Jos all are failed:  one of containers logs as below:     2021-04-16 14:29:19,845 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2021-04-16 14:29:19,895 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2021-04-16 14:29:19,895 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2021-04-16 14:29:19,896 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2021-04-16 14:29:19,896 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1618548626214_0723, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@732c2a62) 2021-04-16 14:29:20,071 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2021-04-16 14:29:24,116 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:29:28,116 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:29:32,117 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:29:36,117 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:29:40,117 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:29:44,118 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:29:48,120 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:29:52,120 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:29:56,121 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:30:00,120 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: dataware-17/10.39.58.15:1872. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-04-16 14:30:03,122 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.net.ConnectException: Call From dataware-11/10.39.58.22 to dataware-17:1872 failed on connection exception: java.net.ConnectException: Connection timed out; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1441) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:246) at com.sun.proxy.$Proxy9.getTask(Unknown Source) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:132) Caused by: java.net.ConnectException: Connection timed out at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:648) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:744) at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557) at org.apache.hadoop.ipc.Client.call(Client.java:1480) ... 4 more       the most important logs is :     Exception running child : java.net.ConnectException: Call From dataware-11/10.39.58.22 to dataware-17:1872 failed on connection exception:     dataware-17 is an application master, dataware-11 is a task, I have already worked on this case two days but can't find root cause.   could you please give me some clues how to solve this kind problems.   ... View more

Hi, everyone   since my company is a group which has many Subsidiary company, so i am thinking about Cloudera Virtual private cluster solution is good for me or not .   all the data save in the base cluster, and if there are new project or business need separate compute resource, then i just arrange a fews machines as a new compute cluster like hive or Flink . i mean this kind architecture is quite clear for me to calculate the resource.   but i don't have any experiences on VPC, does anyone have this kind experience, could you share to me what's your experience or thinking ?   thanks ... View more

today all the impala process crash suddenly, totally is 21 impala daemon, then I restart impala daemon and it's normally just a few second, then crash again. the info logs is like below:   I0114 18:51:33.220736 24116 TAcceptQueueServer.cpp:218] connection_setup_thread_pool_size is set to 2 I0114 18:51:33.959517 23216 ImpaladCatalog.java:200] Adding: TABLE:oyo_behavior_dw.dw_search_log_sort_response version: 7346 size: 79 I0114 18:51:33.959635 23216 ImpaladCatalog.java:200] Adding: CATALOG_SERVICE_ID version: 7346 size: 49 I0114 18:51:33.960319 23216 impala-server.cc:1578] Catalog topic update applied with version: 7346 new min catalog object version: 1366 Wrote minidump to /var/log/impala-minidumps/impalad/73c0cda6-b4cc-4a4c-f8913e88-4ad72a7c.dmp # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f09a2b9a2c0, pid=22656, tid=0x00007f08d9e8f700 # # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libc.so.6+0x1502c0] __memmove_ssse3_back+0x50 # # Core dump written. Default location: /var/log/impalad/core or core.22656 # # An error report file with more information is saved as: # /var/log/impalad/hs_err_pid22656.log # # If you would like to submit a bug report, please visit: # <a href="http://bugreport.java.com/bugreport/crash.jsp" target="_blank">http://bugreport.java.com/bugreport/crash.jsp</a> #   because I have no idea what happened, so I click restart Botton again, I don't remember how many time I restart, because it's  always crashed after restart.  this kind issues happened  two times, it seems this issue also happened one month ago.  I am not sure it's a bug or not.  I have prepared dump file and core file, but how can I upload it? then may be you will find the root cause, thanks.   the below is hs_err_pid22656.log:   # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f09a2b9a2c0, pid=22656, tid=0x00007f08d9e8f700 # # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libc.so.6+0x1502c0] __memmove_ssse3_back+0x50 # # Core dump written. Default location: /var/log/impalad/core or core.22656 # # If you would like to submit a bug report, please visit: # <a href="http://bugreport.java.com/bugreport/crash.jsp" target="_blank">http://bugreport.java.com/bugreport/crash.jsp</a> # --------------- T H R E A D --------------- Current thread is native thread siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000 Registers: RAX=0x000000000c7ec000, RBX=0x0000000000007f33, RCX=0x0000000000007f33, RDX=0x0000000000007f33 RSP=0x00007f08d9e8df68, RBP=0x00007f08d9e8df90, RSI=0x3131353538393300, RDI=0x000000000c7ec000 R8 =0x0000000000000000, R9 =0x0000000000000000, R10=0x0000000000002000, R11=0x0000000000000000 R12=0x3131353538393300, R13=0x0000000010b6c000, R14=0x000000000e2d8d00, R15=0x000000000ba54390 RIP=0x00007f09a2b9a2c0, EFLAGS=0x0000000000010206, CSGSFS=0x8348000000000033, ERR=0x0000000000000000 TRAPNO=0x000000000000000d Top of Stack: (sp=0x00007f08d9e8df68) 0x00007f08d9e8df68: 00000000010ec03a 00007f3300000000 0x00007f08d9e8df78: 000000000c7ec000 00000000110b9010 0x00007f08d9e8df88: 000000000e2ec348 00007f08d9e8dfa0 0x00007f08d9e8df98: 0000000000b7c7f1 00007f08d9e8dfd0 0x00007f08d9e8dfa8: 0000000000b7c816 0000000b00000000 0x00007f08d9e8dfb8: 0000000000000001 000000000db2f290 0x00007f08d9e8dfc8: 000000000f691f00 00007f08d9e8e070 0x00007f08d9e8dfd8: 00000000012fb96d 00007f08d9e8e020 0x00007f08d9e8dfe8: 0000000010277fe8 0025848e00000000 0x00007f08d9e8dff8: 00004519c6189a00 0000000000000031 0x00007f08d9e8e008: 0000000011e634e0 0025848e00000000 0x00007f08d9e8e018: 00004519c6189a00 00007f08d9e8e070 0x00007f08d9e8e028: 00000000012bfbc0 000000080025848e 0x00007f08d9e8e038: 0000000010be45a0 000000000025848e 0x00007f08d9e8e048: 0000000000000ab0 0000000000000000 0x00007f08d9e8e058: 000000000ba54000 00000000012fb530 0x00007f08d9e8e068: 000000000e2d8d00 0000000010b6c000 0x00007f08d9e8e078: 00007f08cc2595ff 0000000000000003 0x00007f08d9e8e088: 000000000e2d8d00 000004000cbadf40 0x00007f08d9e8e098: 00007f08d9e8e1e0 000000000cbadf40 0x00007f08d9e8e0a8: 000000000e2d8e00 0000000010278000 0x00007f08d9e8e0b8: 0000000000000003 00000000d5298327 0x00007f08d9e8e0c8: 000000000fd2f010 0000000000000400 --More--(1%)Instructions: (pc=0x00007f09a2b9a2c0) 0x00007f09a2b9a2a0: fa 90 00 00 00 73 19 48 01 d6 48 01 d7 4c 8d 1d 0x00007f09a2b9a2b0: 1c 44 03 00 49 63 14 93 49 8d 14 13 ff e2 0f 0b 0x00007f09a2b9a2c0: f3 0f 6f 06 49 89 f8 48 83 e7 f0 48 83 c7 10 49 0x00007f09a2b9a2d0: 89 f9 4d 29 c1 4c 29 ca 4c 01 ce 49 89 f1 49 83 Register to memory mapping: RAX=0x000000000c7ec000 is an unknown value RBX=0x0000000000007f33 is an unknown value RCX=0x0000000000007f33 is an unknown value RDX=0x0000000000007f33 is an unknown value RSP=0x00007f08d9e8df68 is an unknown value RBP=0x00007f08d9e8df90 is an unknown value RSI=0x3131353538393300 is an unknown value RDI=0x000000000c7ec000 is an unknown value R8 =0x0000000000000000 is an unknown value R9 =0x0000000000000000 is an unknown value R10=0x0000000000002000 is an unknown value R11=0x0000000000000000 is an unknown value R12=0x3131353538393300 is an unknown value R13=0x0000000010b6c000 is an unknown value R14=0x000000000e2d8d00 is an unknown value R15=0x000000000ba54390 is an unknown value Stack: [0x00007f08d968f000,0x00007f08d9e90000], sp=0x00007f08d9e8df68, free space=8187k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libc.so.6+0x1502c0] __memmove_ssse3_back+0x50 C [impalad+0x77c7f1] impala::AggregateFunctions::StringValGetValue(impala_udf::FunctionContext*, impala_udf::StringVal const&)+0x21 C [impalad+0x77c816] impala::AggregateFunctions::StringValSerializeOrFinalize(impala_udf::FunctionContext*, impala_udf::StringVal const&)+0x16 C [impalad+0xefb96d] impala::AggFnEvaluator::SerializeOrFinalize(impala::Tuple*, impala::SlotDescriptor const&, impala::Tuple*, void*)+0x43d VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap: PSYoungGen total 1835008K, used 343247K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 21% used [0x0000000740000000,0x0000000754f33e68,0x00000007a0000000) from space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000) to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000) ParOldGen total 4194304K, used 37858K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x00000006424f8a50,0x0000000740000000) Metaspace used 35676K, capacity 35974K, committed 36352K, reserved 1081344K class space used 3879K, capacity 4001K, committed 4096K, reserved 1048576K Card table byte_map: [0x00007f098d6fe000,0x00007f098e2ff000] byte_map_base: 0x00007f098a4fe000 Marking Bits: (ParMarkBitMap*) 0x00007f09a60860a0 Begin Bits: [0x00007f097f8e8000, 0x00007f09858e8000) End Bits: [0x00007f09858e8000, 0x00007f098b8e8000) Polling page: 0x00007f09a66ee000 CodeCache: size=245760Kb used=10577Kb max_used=12273Kb free=235182Kb bounds [0x00007f098e6bf000, 0x00007f098f2cf000, 0x00007f099d6bf000] total_blobs=3234 nmethods=2683 adapters=461 compilation: enabled Compilation events (10 events): Event: 49.534 Thread 0x000000000a8ca800 3238 3 org.apache.hadoop.hdfs.protocol.DatanodeInfo$DatanodeInfoBuilder::build (97 bytes) Event: 49.534 Thread 0x000000000a8c9000 3239 3 com.google.protobuf.UnmodifiableLazyStringList$2::next (5 bytes) Event: 49.534 Thread 0x000000000a887000 nmethod 3236 0x00007f098f100750 code [0x00007f098f100920, 0x00007f098f101128] Event: 49.534 Thread 0x000000000a887000 3240 3 com.google.protobuf.UnmodifiableLazyStringList$2::next (13 bytes) Event: 49.534 Thread 0x000000000a8c8000 nmethod 3235 0x00007f098eb21c10 code [0x00007f098eb21de0, 0x00007f098eb225e8] Event: 49.534 Thread 0x000000000a887000 nmethod 3240 0x00007f098ea55bd0 code [0x00007f098ea55d40, 0x00007f098ea56048] Event: 49.534 Thread 0x000000000a887000 3241 3 com.google.protobuf.LazyStringArrayList::get (6 bytes) Event: 49.535 Thread 0x000000000a887000 nmethod 3241 0x00007f098ea55810 code [0x00007f098ea55980, 0x00007f098ea55b28] Event: 49.535 Thread 0x000000000a8c9000 nmethod 3239 0x00007f098ea57350 code [0x00007f098ea574e0, 0x00007f098ea57848] Event: 49.535 Thread 0x000000000a8ca800 nmethod 3238 0x00007f098eb47110 code [0x00007f098eb47280, 0x00007f098eb47648] GC Heap History (8 events): Event: 1.729 GC heap before {Heap before GC invocations=1 (full 0): PSYoungGen total 1835008K, used 534775K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 34% used [0x0000000740000000,0x0000000760a3dc70,0x00000007a0000000) from space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000) to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000) ParOldGen total 4194304K, used 0K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x0000000640000000,0x0000000740000000) Metaspace used 21009K, capacity 21174K, committed 21296K, reserved 1069056K class space used 2350K, capacity 2410K, committed 2432K, reserved 1048576K Event: 1.749 GC heap after Heap after GC invocations=1 (full 0): PSYoungGen total 1835008K, used 17868K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000) from space 262144K, 6% used [0x00000007a0000000,0x00000007a1173250,0x00000007b0000000) to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000) ParOldGen total 4194304K, used 16K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x0000000640004000,0x0000000740000000) Metaspace used 21009K, capacity 21174K, committed 21296K, reserved 1069056K class space used 2350K, capacity 2410K, committed 2432K, reserved 1048576K } Event: 1.749 GC heap before {Heap before GC invocations=2 (full 1): PSYoungGen total 1835008K, used 17868K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000) from space 262144K, 6% used [0x00000007a0000000,0x00000007a1173250,0x00000007b0000000) to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000) ParOldGen total 4194304K, used 16K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x0000000640004000,0x0000000740000000) Metaspace used 21009K, capacity 21174K, committed 21296K, reserved 1069056K class space used 2350K, capacity 2410K, committed 2432K, reserved 1048576K Event: 1.780 GC heap after Heap after GC invocations=2 (full 1): PSYoungGen total 1835008K, used 0K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000) from space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000) to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000) ParOldGen total 4194304K, used 16570K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x000000064102eac0,0x0000000740000000) Metaspace used 21009K, capacity 21174K, committed 21296K, reserved 1069056K class space used 2350K, capacity 2410K, committed 2432K, reserved 1048576K } Event: 48.627 GC heap before {Heap before GC invocations=3 (full 1): PSYoungGen total 1835008K, used 597742K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 38% used [0x0000000740000000,0x00000007647bb868,0x00000007a0000000) from space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000) to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000) ParOldGen total 4194304K, used 16570K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x000000064102eac0,0x0000000740000000) Metaspace used 34984K, capacity 35258K, committed 35456K, reserved 1081344K class space used 3827K, capacity 3935K, committed 3968K, reserved 1048576K Event: 48.653 GC heap after Heap after GC invocations=3 (full 1): PSYoungGen total 1835008K, used 34295K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000) from space 262144K, 13% used [0x00000007b0000000,0x00000007b217df00,0x00000007c0000000) to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000) ParOldGen total 4194304K, used 16642K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x0000000641040ad0,0x0000000740000000) Metaspace used 34984K, capacity 35258K, committed 35456K, reserved 1081344K class space used 3827K, capacity 3935K, committed 3968K, reserved 1048576K } Event: 48.653 GC heap before {Heap before GC invocations=4 (full 2): PSYoungGen total 1835008K, used 34295K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000) from space 262144K, 13% used [0x00000007b0000000,0x00000007b217df00,0x00000007c0000000) to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000) ParOldGen total 4194304K, used 16642K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x0000000641040ad0,0x0000000740000000) Metaspace used 34984K, capacity 35258K, committed 35456K, reserved 1081344K class space used 3827K, capacity 3935K, committed 3968K, reserved 1048576K Event: 48.723 GC heap after Heap after GC invocations=4 (full 2): PSYoungGen total 1835008K, used 0K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000) from space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000) to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000) ParOldGen total 4194304K, used 37858K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 0% used [0x0000000640000000,0x00000006424f8a50,0x0000000740000000) Metaspace used 34984K, capacity 35258K, committed 35456K, reserved 1081344K class space used 3827K, capacity 3935K, committed 3968K, reserved 1048576K } Deoptimization events (10 events): Event: 48.500 Thread 0x00000000116b0800 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00007f098f2b9cf4 method=java.util.AbstractCollection.isEmpty()Z @ 1 Event: 48.501 Thread 0x00000000116b0800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f098edd45c4 method=java.io.DataOutputStream.write(I)V @ 5 Event: 48.504 Thread 0x00000000116b0800 Uncommon trap: reason=predicate action=maybe_recompile pc=0x00007f098f05b814 method=java.util.regex.Pattern$Slice.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 21 Event: 48.565 Thread 0x00000000116b0800 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00007f098ed93d30 method=java.io.DataOutputStream.writeShort(I)V @ 12 Event: 48.611 Thread 0x00000000116b0800 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00007f098ed93d30 method=java.io.DataOutputStream.writeShort(I)V @ 12 Event: 48.748 Thread 0x00000000113a3000 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00007f098ed93d30 method=java.io.DataOutputStream.writeShort(I)V @ 12 Event: 49.379 Thread 0x0000000010d9d800 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00007f098ed93d30 method=java.io.DataOutputStream.writeShort(I)V @ 12 Event: 49.394 Thread 0x0000000011db3000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f098ecabce4 method=java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; @ 98 Event: 49.417 Thread 0x0000000011db3000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f098f17840c method=java.lang.Long.getChars(JI[C)V @ 94 Event: 49.417 Thread 0x0000000011db3000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f098f15c058 method=java.lang.Long.getChars(JI[C)V @ 94 Classes redefined (0 events): No events Internal exceptions (10 events): Event: 0.586 Thread 0x0000000005b00000 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x000000074df5c958) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/runtime/sharedRuntime.cpp, line 605] Event: 0.847 Thread 0x0000000005b00000 Exception <a 'java/lang/NoSuchFieldError': method resolution failed> (0x0000000750246b38) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/methodHandles.cpp, line 1 167] Event: 0.848 Thread 0x0000000005b00000 Exception <a 'java/lang/NoSuchFieldError': method resolution failed> (0x00000007502541f8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/methodHandles.cpp, line 1 167] Event: 1.939 Thread 0x0000000005b00000 Exception <a 'java/lang/NoClassDefFoundError': org/apache/hadoop/hbase/client/ScannerTimeoutException> (0x00000007483e1258) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/ vm/classfile/systemDictionary.cpp, line 199] Event: 1.939 Thread 0x0000000005b00000 Exception <a 'java/lang/NoSuchMethodError': setCaching> (0x00000007483e14b8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jni.cpp, line 1613] Event: 1.945 Thread 0x0000000005b00000 Exception <a 'java/lang/NoSuchMethodError': java.lang.Object.lambda$static$0(Lorg/apache/hadoop/hbase/client/Row;Lorg/apache/hadoop/hbase/client/Row;)I> (0x00000007484ee2e0) thrown at [/HUDSON/workspace/8-2-b uild-linux-amd64/jdk8u181/11358/hotspot/src/sha Event: 2.168 Thread 0x0000000005b00000 Exception <a 'java/lang/ClassNotFoundException': org/apache/impala/util/GlogAppenderBeanInfo> (0x00000007499aaae0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/classf ile/systemDictionary.cpp, line 210] Event: 2.169 Thread 0x0000000005b00000 Exception <a 'java/lang/ClassNotFoundException': org/apache/impala/util/GlogAppenderCustomizer> (0x00000007499eb348) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/clas sfile/systemDictionary.cpp, line 210] Event: 2.169 Thread 0x0000000005b00000 Implicit null exception at 0x00007f098eaf6cf6 to 0x00007f098eaf70b5 Event: 15.697 Thread 0x000000000fbdd000 Implicit null exception at 0x00007f098f21b80c to 0x00007f098f21b82e       ... View more

there is an application to connect kerberos hbase  , but the hbase client won't connect hbase after one day cause client kerberos connection expired .   the java code is below:     System.setProperty("java.security.krb5.conf", "/tmp/krb5.conf"); Configuration conf = HBaseConfiguration.create(); conf.set("hadoop.security.authentication", "kerberos"); conf.set("hbase.security.authentication", "kerberos"); conf.setInt("hbase.client.retries.number", 3); conf.set("hbase.zookeeper.quorum", "10.203.0.26,10.203.0.27,10.203.3.76"); conf.set("hbase.zookeeper.property.clientPort", "2181"); conf.set("hbase.master.kerberos.principal", "hbase/_HOST@OYOHOTELS.CN"); conf.set("hbase.regionserver.kerberos.principal", "hbase/_HOST@OYOHOTELS.CN"); UserGroupInformation.setConfiguration(conf); UserGroupInformation.loginUserFromKeytab("hbase@OYOHOTELS.CN", "/tmp/hbase.keytab"); Connection conn = ConnectionFactory.createConnection(conf);       developer has checked the hbase client code, and find there is a parameter :  hadoop.kerberos.keytab.login.autorenewal.enabled, if this parameter set true in core-site.xml , then it will re-login after the kerberos expired.        /** * * <a href="{@docRoot}/../hadoop-project-dist/hadoop-common/core-default.xml"> * core-default.xml</a> */ public static final String HADOOP_KERBEROS_KEYTAB_LOGIN_AUTORENEWAL_ENABLED = "hadoop.kerberos.keytab.login.autorenewal.enabled"; /** Default value for HADOOP_KERBEROS_KEYTAB_LOGIN_AUTORENEWAL_ENABLED. */ public static final boolean HADOOP_KERBEROS_KEYTAB_LOGIN_AUTORENEWAL_ENABLED_DEFAULT = false;   my question is : I haven't found this parameter in hbase documents or CDH documents , this parameter can resolve  this issue or not ?   if not, how to resolve hbase client kerberos connection issue? ... View more

Hi, everyone  I have tested HBase client by java code, and found if I am going to comment out the /etc/hosts , hbase client can't connect HBase server envv .    after test many times, I can sure hbase must depend on /etc/hosts.    anyone can give any advises there are any other solution if I comment out /etc/hosts.     thanks . ... View more