Member since
02-23-2016
22
Posts
0
Kudos Received
0
Solutions
03-22-2017
03:15 AM
Thank you! It's really because storm workers are suffering GC pauses! We will check the code of storm workers.
... View more
03-21-2017
08:55 PM
I'm sorry, I mistake the storm log. There are many IOExeception in kudu client log, such as "[ERROR 2017-03-22 10:30:21 o.a.k.c.TabletClient:791 New I/O worker #1] [Peer 772d8abf66c24789a51ba852aea1b478] Unexpected exception from downstream on [id: 0x2b9826d2, /11.131.120.17:55922 => /11.173.197.31:7050] java.io.IOException: Broken pipe"
... View more
03-21-2017
07:39 PM
Write speed decreases from 200K rows/second to 5K rows/second. There are many warnings "error 104"/"error 108" in trace log. The data in kudu is not available.
... View more
03-21-2017
06:53 PM
Here are some trace messages: W0322 09:40:06.534153 129361 negotiation.cc:240] Failed RPC negotiation. Trace: 0322 09:39:51.533481 (+ 0us) reactor.cc:378] Submitting negotiation task for server connection from 11.136.156.50:39299 0322 09:39:51.533517 (+ 36us) sasl_server.cc:230] Waiting for connection header 0322 09:39:57.673309 (+6139792us) sasl_server.cc:238] Connection header received 0322 09:39:57.673309 (+ 0us) sasl_server.cc:181] Waiting for next SASL message... 0322 09:40:06.534117 (+8860808us) negotiation.cc:231] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 11.136.156.50:39299 Metrics: {"negotiator.queue_time_us":8} W0322 09:40:16.245877 164604 connection.cc:464] server connection from 11.136.156.55:43035 recv error: Network error: recv error: Connection reset by peer (error 104) W0322 09:40:16.246330 164604 connection.cc:378] Connection torn down before Call kudu.tserver.TabletServerService.Write from 11.136.156.55:43035 (ReqId={client: 574a7048c3824c869a1e553825845ba6, seq_no=467150, attempt_no=1}) could send its response ohter trace messages: W0322 09:27:45.480054 101702 negotiation.cc:240] Failed RPC negotiation. Trace: 0322 09:27:31.439139 (+ 0us) reactor.cc:378] Submitting negotiation task for server connection from 11.1 31.120.17:41515 0322 09:27:31.439526 (+ 387us) sasl_server.cc:230] Waiting for connection header 0322 09:27:32.087993 (+648467us) sasl_server.cc:238] Connection header received 0322 09:27:32.087994 (+ 1us) sasl_server.cc:181] Waiting for next SASL message... 0322 09:27:45.479950 (+13391956us) sasl_server.cc:295] SASL Server: Received NEGOTIATE request from client 0322 09:27:45.479964 (+ 14us) sasl_server.cc:341] Sent NEGOTIATE response 0322 09:27:45.479967 (+ 3us) sasl_server.cc:181] Waiting for next SASL message... 0322 09:27:45.480007 (+ 40us) negotiation.cc:231] Negotiation complete: Network error: Server connection negotiation failed: server connection from 11.131.120.17:41515: BlockingRecv error: Recv() got EOF from remo te (error 108) Metrics: {"negotiator.queue_time_us":352,"thread_start_us":344,"threads_started":1}
... View more
03-20-2017
10:29 PM
Sometimes the WARNING "Negotiation complete: Network error: Server connection negotiation failed: server connection from 11.137.128.14:20391: BlockingRecv error: Recv() got EOF from remote (error 108)" occured when a lot of rows(about 200k rows/second) were inserted into kudu by storm. Then I restarted the storm job, the inserting job would be OK. No WARN/ERROR log appeared in storm.
... View more
Labels:
- Labels:
-
Apache Kudu
02-23-2016
08:10 PM
Are there any wrong about the C Flags? Compiler Flags: -Wall -Wno-sign-compare -Wno-unknown-pragmas -pthread -fno-strict-aliasing -Wno-deprecated -DBOOST_DATE_TIME_POSIX_TIME_STD_CONFIG -g -Wno-unused-local-typedefs -O3 -DNDEBUG -gdwarf-2 -fverbose-asm -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
... View more
02-23-2016
07:32 PM
Hi all, Why the impalad from the CDH parcel can’t be replaced the one built by myself? When I build impala with “-build_static_libs”, the size of impalad built is about 260M, but the one from the CDH parcel is 23M When I use the “-build_shared_libs”, the size of impalad built is about 38M, but it have different *.so with the one from the CDH parcel. Thank you for your time on this. I hope your response as soon as possible. Best wishes. Below is the information about “ldd” With “-build_shared_libs”: linux-vdso.so.1 => (0x00007fff2a9ff000) libjsig.so => /usr/java/jdk1.7.0_79/jre/lib/amd64/libjsig.so (0x00007f713d918000) libCatalog.so => /home/test/opt/impala/be/build/debug/catalog/libCatalog.so (0x00007f713d5a3000) libCodeGen.so => /home/test/opt/impala/be/build/debug/codegen/libCodeGen.so (0x00007f713d2d4000) libCommon.so => /home/test/opt/impala/be/build/debug/common/libCommon.so (0x00007f713d094000) libExec.so => /home/test/opt/impala/be/build/debug/exec/libExec.so (0x00007f713c96f000) libExprs.so => /home/test/opt/impala/be/build/debug/exprs/libExprs.so (0x00007f713c4dc000) libGlobalFlags.so => /home/test/opt/impala/be/build/debug/common/libGlobalFlags.so (0x00007f713c2d9000) libImpalaThrift.so => /home/test/opt/impala/be/build/debug/thrift/libImpalaThrift.so (0x00007f713badf000) libResourceBroker.so => /home/test/opt/impala/be/build/debug/resourcebroker/libResourceBroker.so (0x00007f713b76f000) libRpc.so => /home/test/opt/impala/be/build/debug/rpc/libRpc.so (0x00007f713b3bd000) libRuntime.so => /home/test/opt/impala/be/build/debug/runtime/libRuntime.so (0x00007f713abdf000) libService.so => /home/test/opt/impala/be/build/debug/service/libService.so (0x00007f713a570000) libStatestore.so => /home/test/opt/impala/be/build/debug/statestore/libStatestore.so (0x00007f713a048000) libTestUtil.so => /home/test/opt/impala/be/build/debug/testutil/libTestUtil.so (0x00007f7139d97000) libThriftSaslTransport.so => /home/test/opt/impala/be/build/debug/transport/libThriftSaslTransport.so (0x00007f7139b46000) libUdf.so => /home/test/opt/impala/be/build/debug/udf/libUdf.so (0x00007f7139915000) libUtil.so => /home/test/opt/impala/be/build/debug/util/libUtil.so (0x00007f713942c000) libboost_thread.so.1.55.0 => /usr/local/lib/libboost_thread.so.1.55.0 (0x00007f7139213000) libboost_regex.so.1.55.0 => /usr/local/lib/libboost_regex.so.1.55.0 (0x00007f7138f26000) libboost_system.so.1.55.0 => /usr/local/lib/libboost_system.so.1.55.0 (0x00007f7138d23000) libboost_filesystem.so.1.55.0 => /usr/local/lib/libboost_filesystem.so.1.55.0 (0x00007f7138b0c000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f7138901000) libjvm.so => /usr/java/jdk1.7.0_79/jre/lib/amd64/server/libjvm.so (0x00007f7137a87000) libhdfs.so.0.0.0 => /home/test/opt/impala/thirdparty/hadoop-2.6.0-cdh5.4.1/lib/native/libhdfs.so.0.0.0 (0x00007f7137877000) libgutil.so => /home/test/opt/impala/thirdparty/gutil/libgutil.so (0x00007f71375c9000) libz.so.1 => /lib64/libz.so.1 (0x00007f71373b3000) libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f71371a1000) librt.so.1 => /lib64/librt.so.1 (0x00007f7136f99000) libboost_date_time.so.1.55.0 => /usr/local/lib/libboost_date_time.so.1.55.0 (0x00007f7136d88000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7136b6a000) libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f71368fe000) libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f713651b000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f7136214000) libm.so.6 => /lib64/libm.so.6 (0x00007f7135f90000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f7135d7a000) libc.so.6 => /lib64/libc.so.6 (0x00007f71359e5000) /lib64/ld-linux-x86-64.so.2 (0x00007f713db1d000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f71357a0000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f71354b9000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f71352b5000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f7135088000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f7134e7d000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f7134c79000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f7134a5f000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f7134840000) With “-build_static_libs”: linux-vdso.so.1 => (0x00007fffaf1b8000) /usr/java/jdk1.7.0_79//jre/lib/amd64/server/libjsig.so (0x00007f82dec0e000) libboost_thread.so.1.55.0 => /usr/local/lib/libboost_thread.so.1.55.0 (0x00007f82de9f6000) libboost_regex.so.1.55.0 => /usr/local/lib/libboost_regex.so.1.55.0 (0x00007f82de709000) libboost_system.so.1.55.0 => /usr/local/lib/libboost_system.so.1.55.0 (0x00007f82de506000) libboost_filesystem.so.1.55.0 => /usr/local/lib/libboost_filesystem.so.1.55.0 (0x00007f82de2ef000) libboost_date_time.so.1.55.0 => /usr/local/lib/libboost_date_time.so.1.55.0 (0x00007f82de0de000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f82dded3000) libz.so.1 => /lib64/libz.so.1 (0x00007f82ddcbc000) libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f82ddaab000) libjvm.so => /usr/java/jdk1.7.0_79/jre/lib/amd64/server/libjvm.so (0x00007f82dcc31000) librt.so.1 => /lib64/librt.so.1 (0x00007f82dca28000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f82dc80b000) libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f82dc59f000) libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f82dc1bb000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f82dbeb5000) libm.so.6 => /lib64/libm.so.6 (0x00007f82dbc31000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f82dba1a000) libc.so.6 => /lib64/libc.so.6 (0x00007f82db686000) /lib64/ld-linux-x86-64.so.2 (0x00007f82dee13000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f82db441000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f82db15a000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f82daf56000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f82dad29000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f82dab1e000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f82da91a000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f82da700000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f82da4e1000) from the CDH parcels linux-vdso.so.1 => (0x00007fff97b9d000) libjsig.so => not found libdl.so.2 => /lib64/libdl.so.2 (0x00000032f6800000) libz.so.1 => /lib64/libz.so.1 (0x00000032f7400000) libbz2.so.1 => /lib64/libbz2.so.1 (0x00000032ffc00000) libjvm.so => not found librt.so.1 => /lib64/librt.so.1 (0x00000032f7800000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000032f7000000) libssl.so.10 => /usr/lib64/libssl.so.10 (0x00000032fe800000) libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00000032fa400000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000032f8000000) libm.so.6 => /lib64/libm.so.6 (0x00000032f7c00000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000032f8c00000) libc.so.6 => /lib64/libc.so.6 (0x00000032f6c00000) /lib64/ld-linux-x86-64.so.2 (0x00000032f6400000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00000032fd800000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00000032fc800000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00000032fa800000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00000032fd000000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00000032fd400000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00000032fcc00000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00000032f8800000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00000032f8400000)
... View more
Labels: