Support Questions
Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Advanced Search

Impala catalogd crash with Kerberos-based security enabled

Highlighted

Impala catalogd crash with Kerberos-based security enabled

valtri
Contributor

Created ‎10-18-2015 02:21 PM

Catalogd crashed when trying Kerberos security. I'm sending the report:

 

packages+versions:

impala	2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy-cdh5.4.7
impala-catalog	2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy-cdh5.4.7
impala-dbg	2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy-cdh5.4.7
heimdal-clients	1.6~git20120403+dfsg1-2
openjdk-7-jre-headless:amd64	7u79-2.5.6-1~deb7u1
libsasl2-2:amd64	2.1.25.dfsg1-6+deb7u1
libsasl2-modules:amd64	2.1.25.dfsg1-6+deb7u1

/etc/default/impala:

IMPALA_CATALOG_SERVICE_HOST=myriad12.zcu.cz
IMPALA_STATE_STORE_HOST=myriad4.zcu.cz
IMPALA_STATE_STORE_PORT=24000
IMPALA_BACKEND_PORT=22000
IMPALA_LOG_DIR=/var/log/impala

IMPALA_CATALOG_ARGS=" \
    -log_dir=${IMPALA_LOG_DIR} \
    -state_store_host=${IMPALA_STATE_STORE_HOST} \
    -principal=impala/myriad12.zcu.cz@ZCU.CZ \
    -keytab_file /etc/security/keytab/impala.service.keytab"
IMPALA_STATE_STORE_ARGS=" \
    -log_dir=${IMPALA_LOG_DIR} \
    -state_store_port=${IMPALA_STATE_STORE_PORT} \
    -principal=impala/myriad4.zcu.cz@ZCU.CZ \
    -keytab_file /etc/security/keytab/impala.service.keytab"
IMPALA_SERVER_ARGS=" \
    -log_dir=${IMPALA_LOG_DIR} \
    -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \
    -state_store_port=${IMPALA_STATE_STORE_PORT} \
    -use_statestore \
    -state_store_host=${IMPALA_STATE_STORE_HOST} \
    -be_port=${IMPALA_BACKEND_PORT}"

ENABLE_CORE_DUMPS=true

Backtrace:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/impala/sbin/catalogd -log_dir=/var/log/impala -state_store_host=myriad'.
Program terminated with signal 6, Aborted.
#0  0x00007f8d0b6cc165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007f8d0b6cc165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f8d0b6cf3e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f8d0d3e66f9 in os::abort(bool) () from /usr/lib/jvm/java-1.7.0-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#3  0x00007f8d0d55fa1f in VMError::report_and_die() ()
   from /usr/lib/jvm/java-1.7.0-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#4  0x00007f8d0d3ef79b in JVM_handle_linux_signal () from /usr/lib/jvm/java-1.7.0-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x00007f8d0b719db1 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000000000b14bfb in length (__s=0x3030303432 <Address 0x3030303432 out of bounds>)
    at /usr/include/c++/4.7/bits/char_traits.h:261
#8  assign (__s=0x3030303432 <Address 0x3030303432 out of bounds>, this=0x5b95fa0)
    at /usr/include/c++/4.7/bits/basic_string.h:1128
#9  operator= (__s=0x3030303432 <Address 0x3030303432 out of bounds>, this=0x5b95fa0)
    at /usr/include/c++/4.7/bits/basic_string.h:552
#10 sasl::TSaslClient::evaluateChallengeOrResponse (this=0x5b95f80, challenge=<optimized out>, len=<optimized out>, 
    resLen=0x7ffe2931e0cc)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/transport/TSasl.cpp:147
#11 0x0000000000b14752 in apache::thrift::transport::TSaslClientTransport::handleSaslStartMessage (this=0x692ec40)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/transport/TSaslClientTransport.cpp:48
#12 0x0000000000b186cf in apache::thrift::transport::TSaslTransport::open (this=0x692ec40)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/transport/TSaslTransport.cpp:95
#13 0x0000000000c0a3b0 in impala::ThriftClientImpl::Open (this=this@entry=0x4eb7c00)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/rpc/thrift-client.cc:39
#14 0x0000000000c0a4f8 in impala::ThriftClientImpl::OpenWithRetry (this=0x4eb7c00, num_tries=10, wait_ms=3000)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/rpc/thrift-client.cc:52
#15 0x0000000000a5373a in impala::ClientCacheHelper::CreateClient(impala::TNetworkAddress const&, boost::function<impala::Thrift
ClientImpl* (impala::TNetworkAddress const&, void**)>, void**) (this=this@entry=0x5abd100, address=..., factory_method=..., 
    client_key=client_key@entry=0x7ffe2931e6e8)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/runtime/client-cache.cc:109
#16 0x0000000000a53dfb in impala::ClientCacheHelper::GetClient(impala::TNetworkAddress const&, boost::function<impala::ThriftClientImpl* (impala::TNetworkAddress const&, void**)>, void**) (this=0x5abd100, address=..., factory_method=..., 
    client_key=0x7ffe2931e6e8)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/runtime/client-cache.cc:64
#17 0x0000000000b0ee9f in impala::ClientCache<impala::StatestoreServiceClient>::GetClient (this=0x5abd100, address=..., 
    iface=0x7ffe2931e6e8)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/runtime/client-cache.h:329
#18 0x0000000000b0eeff in impala::ClientConnection<impala::StatestoreServiceClient>::ClientConnection (this=<optimized out>, 
    client_cache=<optimized out>, address=..., status=0x7ffe2931e690)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/runtime/client-cache.h:211
#19 0x0000000000b08576 in impala::StatestoreSubscriber::Register (this=this@entry=0x79c1200)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/statestore/statestore-subscriber.cc:143
#20 0x0000000000b0a239 in impala::StatestoreSubscriber::Start (this=0x79c1200)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/statestore/statestore-subscriber.cc:204
#21 0x000000000089761c in impala::CatalogServer::Start (this=0x7ffe2931ed20)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/catalog/catalog-server.cc:182
#22 0x000000000087aa31 in main (argc=<optimized out>, argv=<optimized out>)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.4.7-Packaging-Impala-2015-09-17_01-59-47/impala-2.2.0+cdh5.4.7+0-1.cdh5.4.7.p0.3~wheezy/be/src/catalog/catalogd-main.cc:75

Cheers,

  František

Reply
1,125 Views
0 Kudos
4 REPLIES 4

Re: Impala catalogd crash with Kerberos-based security enabled

valtri
Contributor

Created ‎11-06-2015 03:45 AM

Reproduced with Impala 2.2.0/CDH 5.4.8 and reported: IMPALA-2645

Reply
1,094 Views
0 Kudos
Highlighted

Re: Impala catalogd crash with Kerberos-based security enabled

valtri
Contributor

Created ‎12-03-2015 08:08 AM

Still crashing with Impala 2.3.0/CDH 5.5.0.

Reply
1,067 Views
0 Kudos
Highlighted

Re: Impala catalogd crash with Kerberos-based security enabled

valtri
Contributor

Created ‎12-17-2015 05:53 AM

I've tried also with the Oracle JDK (now with Impala 2.3.0/CDH 5.5.1).

 

Packages:

heimdal-clients 1.6~git20120403+dfsg1-2
impala  2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy-cdh5.5.1
impala-dbg      2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy-cdh5.5.1
impala-server   2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy-cdh5.5.1
libsasl2-2:amd64        2.1.25.dfsg1-6+deb7u1
libsasl2-modules:amd64  2.1.25.dfsg1-6+deb7u1
oracle-java8-installer  8u66+8u65arm-1~webupd8~1
oracle-java8-unlimited-jce-policy       8u66+8u65arm-1~webupd8~1

The crash looks the same (for impalad):

(gdb) bt
#0  0x00007f58abd82165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f58abd853e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f58ad9f6795 in os::abort(bool) () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
#3  0x00007f58adb94e23 in VMError::report_and_die() () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
#4  0x00007f58ad9fbfbf in JVM_handle_linux_signal () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
#5  0x00007f58ad9f2753 in signalHandler(int, siginfo*, void*) ()
   from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7  0x00007f58abdcfdb1 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#8  0x0000000000b5194b in length (__s=0x3030303432 <Address 0x3030303432 out of bounds>)
    at /usr/include/c++/4.7/bits/char_traits.h:261
#9  assign (__s=0x3030303432 <Address 0x3030303432 out of bounds>, this=0x51f0260)
    at /usr/include/c++/4.7/bits/basic_string.h:1128
#10 operator= (__s=0x3030303432 <Address 0x3030303432 out of bounds>, this=0x51f0260)
    at /usr/include/c++/4.7/bits/basic_string.h:552
#11 sasl::TSaslClient::evaluateChallengeOrResponse (this=0x51f0240, challenge=<optimized out>, len=<optimized out>, 
    resLen=0x7fff2f05629c)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSasl.cpp:151
#12 0x0000000000b514a3 in apache::thrift::transport::TSaslClientTransport::handleSaslStartMessage (this=0x64db080)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSaslClientTransport.cpp:48
#13 0x0000000000b554af in apache::thrift::transport::TSaslTransport::open (this=0x64db080)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSaslTransport.cpp:95
#14 0x0000000000c7274a in impala::ThriftClientImpl::Open (this=this@entry=0x5bab700)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/rpc/thrift-client.cc:39
#15 0x0000000000c72888 in impala::ThriftClientImpl::OpenWithRetry (this=0x5bab700, num_tries=10, wait_ms=3000)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/rpc/thrift-client.cc:52
#16 0x0000000000a8a4da in impala::ClientCacheHelper::CreateClient(impala::TNetworkAddress const&, boost::function<impala::ThriftClientImpl* (impala::TNetworkAddress const&, void**)>, void**) (this=this@entry=0x5dbdb00, address=..., 
    factory_method=..., client_key=client_key@entry=0x7fff2f0568a8)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/client-cache.cc:108
#17 0x0000000000a8ab8b in impala::ClientCacheHelper::GetClient(impala::TNetworkAddress const&, boost::function<impala::ThriftClientImpl* (impala::TNetworkAddress const&, void**)>, void**) (this=0x5dbdb00, address=..., factory_method=..., 
    client_key=0x7fff2f0568a8)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/client-cache.cc:63
#18 0x0000000000b4b65f in impala::ClientCache<impala::StatestoreServiceClient>::GetClient (this=0x5dbdb00, address=..., 
    iface=0x7fff2f0568a8)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/client-cache.h:320
#19 0x0000000000b4b6bf in impala::ClientConnection<impala::StatestoreServiceClient>::ClientConnection (this=<optimized out>, 
    client_cache=<optimized out>, address=..., status=0x7fff2f056860)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/client-cache.h:196
#20 0x0000000000b44c26 in impala::StatestoreSubscriber::Register (this=this@entry=0x5d2de00)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/statestore/statestore-subscriber.cc:139
#21 0x0000000000b46779 in impala::StatestoreSubscriber::Start (this=0x5d2de00)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/statestore/statestore-subscriber.cc:191
#22 0x0000000000a5b21d in impala::ExecEnv::StartServices (this=0x7fff2f056d70)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/runtime/exec-env.cc:400
#23 0x0000000000897fea in main (argc=<optimized out>, argv=<optimized out>)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+---Type <return> to continue, or q <return> to quit---
cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/service/impalad-main.cc:78
(gdb) f 11
#11 sasl::TSaslClient::evaluateChallengeOrResponse (this=0x51f0240, challenge=<optimized out>, len=<optimized out>, 
    resLen=0x7fff2f05629c)
    at /data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSasl.cpp:151
151	/data/jenkins/workspace/generic-package-debian64-7-0-impala/CDH5.5.1-Packaging-Impala-2015-12-02_10-27-37/impala-2.3.0+cdh5.5.1+0-1.cdh5.5.1.p0.17~wheezy/be/src/transport/TSasl.cpp: No such file or directory.
(gdb) p *this
$1 = {<sasl::TSasl> = {_vptr.TSasl = 0x1d41cd0, authComplete = false, conn = 0x5528400}, clientStarted = true, chosenMech = {
    static npos = <optimized out>, 
    _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
      _M_p = 0x30c4298 "GSSAPI"}}, mechList = {static npos = <optimized out>, 
    _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
      _M_p = 0x30c4298 "GSSAPI"}}}
Reply
1,040 Views
0 Kudos
Highlighted

Impala crashing with security

valtri
Contributor

Created ‎12-17-2015 05:55 AM

 
Reply
1,039 Views
0 Kudos
Don't have an account?
Register
Announcements
Product Announcements
[ANNOUNCE] New Cloudera JDBC 2.6.20 Driver for Apache Impala Released
Product Announcements
Transition to private repositories for CDH, HDP and HDF
Product Announcements
[ANNOUNCE] New Applied ML Research from Cloudera Fast Forward: Few-Shot Text Classification
Product Announcements
[ANNOUNCE] New JDBC 2.6.13 Driver for Apache Hive Released
Product Announcements
[ANNOUNCE] Refreshed Research from Cloudera Fast Forward: Semantic Image Search and Federated Learning
View More Announcements