Created 10-18-2015 02:21 PM
Catalogd crashed when trying Kerberos security. I'm sending the report:
packages+versions:
impala 2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy-cdh5.4.7 impala-catalog 2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy-cdh5.4.7 impala-dbg 2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy-cdh5.4.7 heimdal-clients 1.6~git20120403+dfsg1-2 openjdk-7-jre-headless:amd64 7u79-2.5.6-1~deb7u1 libsasl2-2:amd64 2.1.25.dfsg1-6+deb7u1 libsasl2-modules:amd64 2.1.25.dfsg1-6+deb7u1
/etc/default/impala:
IMPALA_CATALOG_SERVICE_HOST=myriad12.zcu.cz IMPALA_STATE_STORE_HOST=myriad4.zcu.cz IMPALA_STATE_STORE_PORT=24000 IMPALA_BACKEND_PORT=22000 IMPALA_LOG_DIR=/var/log/impala IMPALA_CATALOG_ARGS=" \ -log_dir=${IMPALA_LOG_DIR} \ -state_store_host=${IMPALA_STATE_STORE_HOST} \ -principal=impala/myriad12.zcu.cz@ZCU.CZ \ -keytab_file /etc/security/keytab/impala.service.keytab" IMPALA_STATE_STORE_ARGS=" \ -log_dir=${IMPALA_LOG_DIR} \ -state_store_port=${IMPALA_STATE_STORE_PORT} \ -principal=impala/myriad4.zcu.cz@ZCU.CZ \ -keytab_file /etc/security/keytab/impala.service.keytab" IMPALA_SERVER_ARGS=" \ -log_dir=${IMPALA_LOG_DIR} \ -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \ -state_store_port=${IMPALA_STATE_STORE_PORT} \ -use_statestore \ -state_store_host=${IMPALA_STATE_STORE_HOST} \ -be_port=${IMPALA_BACKEND_PORT}" ENABLE_CORE_DUMPS=true
Backtrace:
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/lib/impala/sbin/catalogd -log_dir=/var/log/impala -state_store_host=myriad'. Program terminated with signal 6, Aborted. #0 0x00007f8d0b6cc165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007f8d0b6cc165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f8d0b6cf3e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f8d0d3e66f9 in os::abort(bool) () from /usr/lib/jvm/java-1.7.0-openjdk-amd64/jre/lib/amd64/server/libjvm.so #3 0x00007f8d0d55fa1f in VMError::report_and_die() () from /usr/lib/jvm/java-1.7.0-openjdk-amd64/jre/lib/amd64/server/libjvm.so #4 0x00007f8d0d3ef79b in JVM_handle_linux_signal () from /usr/lib/jvm/java-1.7.0-openjdk-amd64/jre/lib/amd64/server/libjvm.so #5 <signal handler called> #6 0x00007f8d0b719db1 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000b14bfb in length (__s=0x3030303432 <Address 0x3030303432 out of bounds>) at /usr/include/c++/4.7/bits/char_traits.h:261 #8 assign (__s=0x3030303432 <Address 0x3030303432 out of bounds>, this=0x5b95fa0) at /usr/include/c++/4.7/bits/basic_string.h:1128 #9 operator= (__s=0x3030303432 <Address 0x3030303432 out of bounds>, this=0x5b95fa0) at /usr/include/c++/4.7/bits/basic_string.h:552 #10 sasl::TSaslClient::evaluateChallengeOrResponse (this=0x5b95f80, challenge=<optimized out>, len=<optimized out>, resLen=0x7ffe2931e0cc) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/transport/TSasl.cpp:147 #11 0x0000000000b14752 in apache::thrift::transport::TSaslClientTransport::handleSaslStartMessage (this=0x692ec40) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/transport/TSaslClientTransport.cpp:48 #12 0x0000000000b186cf in apache::thrift::transport::TSaslTransport::open (this=0x692ec40) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/transport/TSaslTransport.cpp:95 #13 0x0000000000c0a3b0 in impala::ThriftClientImpl::Open (this=this@entry=0x4eb7c00) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/rpc/thrift-client.cc:39 #14 0x0000000000c0a4f8 in impala::ThriftClientImpl::OpenWithRetry (this=0x4eb7c00, num_tries=10, wait_ms=3000) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/rpc/thrift-client.cc:52 #15 0x0000000000a5373a in impala::ClientCacheHelper::CreateClient(impala::TNetworkAddress const&, boost::function<impala::Thrift ClientImpl* (impala::TNetworkAddress const&, void**)>, void**) (this=this@entry=0x5abd100, address=..., factory_method=..., client_key=client_key@entry=0x7ffe2931e6e8) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/runtime/client-cache.cc:109 #16 0x0000000000a53dfb in impala::ClientCacheHelper::GetClient(impala::TNetworkAddress const&, boost::function<impala::ThriftClientImpl* (impala::TNetworkAddress const&, void**)>, void**) (this=0x5abd100, address=..., factory_method=..., client_key=0x7ffe2931e6e8) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/runtime/client-cache.cc:64 #17 0x0000000000b0ee9f in impala::ClientCache<impala::StatestoreServiceClient>::GetClient (this=0x5abd100, address=..., iface=0x7ffe2931e6e8) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/runtime/client-cache.h:329 #18 0x0000000000b0eeff in impala::ClientConnection<impala::StatestoreServiceClient>::ClientConnection (this=<optimized out>, client_cache=<optimized out>, address=..., status=0x7ffe2931e690) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/runtime/client-cache.h:211 #19 0x0000000000b08576 in impala::StatestoreSubscriber::Register (this=this@entry=0x79c1200) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/statestore/statestore-subscriber.cc:143 #20 0x0000000000b0a239 in impala::StatestoreSubscriber::Start (this=0x79c1200) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/statestore/statestore-subscriber.cc:204 #21 0x000000000089761c in impala::CatalogServer::Start (this=0x7ffe2931ed20) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/catalog/catalog-server.cc:182 #22 0x000000000087aa31 in main (argc=<optimized out>, argv=<optimized out>) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/catalog/catalogd-main.cc:75
Cheers,
František
Created 11-06-2015 03:45 AM
Reproduced with Impala 2.2.0/CDH 5.4.8 and reported: IMPALA-2645
Created 12-03-2015 08:08 AM
Still crashing with Impala 2.3.0/CDH 5.5.0.
Created 12-17-2015 05:53 AM
I've tried also with the Oracle JDK (now with Impala 2.3.0/CDH 5.5.1).
Packages:
heimdal-clients 1.6~git20120403+dfsg1-2 impala 2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy-cdh5.5.1 impala-dbg 2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy-cdh5.5.1 impala-server 2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy-cdh5.5.1 libsasl2-2:amd64 2.1.25.dfsg1-6+deb7u1 libsasl2-modules:amd64 2.1.25.dfsg1-6+deb7u1 oracle-java8-installer 8u66+8u65arm-1~webupd8~1 oracle-java8-unlimited-jce-policy 8u66+8u65arm-1~webupd8~1
The crash looks the same (for impalad):
(gdb) bt #0 0x00007f58abd82165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f58abd853e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f58ad9f6795 in os::abort(bool) () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so #3 0x00007f58adb94e23 in VMError::report_and_die() () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so #4 0x00007f58ad9fbfbf in JVM_handle_linux_signal () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so #5 0x00007f58ad9f2753 in signalHandler(int, siginfo*, void*) () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so #6 <signal handler called> #7 0x00007f58abdcfdb1 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x0000000000b5194b in length (__s=0x3030303432 <Address 0x3030303432 out of bounds>) at /usr/include/c++/4.7/bits/char_traits.h:261 #9 assign (__s=0x3030303432 <Address 0x3030303432 out of bounds>, this=0x51f0260) at /usr/include/c++/4.7/bits/basic_string.h:1128 #10 operator= (__s=0x3030303432 <Address 0x3030303432 out of bounds>, this=0x51f0260) at /usr/include/c++/4.7/bits/basic_string.h:552 #11 sasl::TSaslClient::evaluateChallengeOrResponse (this=0x51f0240, challenge=<optimized out>, len=<optimized out>, resLen=0x7fff2f05629c) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSasl.cpp:151 #12 0x0000000000b514a3 in apache::thrift::transport::TSaslClientTransport::handleSaslStartMessage (this=0x64db080) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSaslClientTransport.cpp:48 #13 0x0000000000b554af in apache::thrift::transport::TSaslTransport::open (this=0x64db080) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSaslTransport.cpp:95 #14 0x0000000000c7274a in impala::ThriftClientImpl::Open (this=this@entry=0x5bab700) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/rpc/thrift-client.cc:39 #15 0x0000000000c72888 in impala::ThriftClientImpl::OpenWithRetry (this=0x5bab700, num_tries=10, wait_ms=3000) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/rpc/thrift-client.cc:52 #16 0x0000000000a8a4da in impala::ClientCacheHelper::CreateClient(impala::TNetworkAddress const&, boost::function<impala::ThriftClientImpl* (impala::TNetworkAddress const&, void**)>, void**) (this=this@entry=0x5dbdb00, address=..., factory_method=..., client_key=client_key@entry=0x7fff2f0568a8) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/client-cache.cc:108 #17 0x0000000000a8ab8b in impala::ClientCacheHelper::GetClient(impala::TNetworkAddress const&, boost::function<impala::ThriftClientImpl* (impala::TNetworkAddress const&, void**)>, void**) (this=0x5dbdb00, address=..., factory_method=..., client_key=0x7fff2f0568a8) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/client-cache.cc:63 #18 0x0000000000b4b65f in impala::ClientCache<impala::StatestoreServiceClient>::GetClient (this=0x5dbdb00, address=..., iface=0x7fff2f0568a8) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/client-cache.h:320 #19 0x0000000000b4b6bf in impala::ClientConnection<impala::StatestoreServiceClient>::ClientConnection (this=<optimized out>, client_cache=<optimized out>, address=..., status=0x7fff2f056860) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/client-cache.h:196 #20 0x0000000000b44c26 in impala::StatestoreSubscriber::Register (this=this@entry=0x5d2de00) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/statestore/statestore-subscriber.cc:139 #21 0x0000000000b46779 in impala::StatestoreSubscriber::Start (this=0x5d2de00) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/statestore/statestore-subscriber.cc:191 #22 0x0000000000a5b21d in impala::ExecEnv::StartServices (this=0x7fff2f056d70) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/exec-env.cc:400 #23 0x0000000000897fea in main (argc=<optimized out>, argv=<optimized out>) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+---Type <return> to continue, or q <return> to quit--- cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/service/impalad-main.cc:78 (gdb) f 11 #11 sasl::TSaslClient::evaluateChallengeOrResponse (this=0x51f0240, challenge=<optimized out>, len=<optimized out>, resLen=0x7fff2f05629c) at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSasl.cpp:151 151 /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSasl.cpp: No such file or directory. (gdb) p *this $1 = {<sasl::TSasl> = {_vptr.TSasl = 0x1d41cd0, authComplete = false, conn = 0x5528400}, clientStarted = true, chosenMech = { static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x30c4298 "GSSAPI"}}, mechList = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x30c4298 "GSSAPI"}}}
Created 12-17-2015 05:55 AM