Kerberos Authentication Failure : Catalog Server Unable to Connect to Statestore Port

[#1]

F0303 09:59:04.650674 32117 catalogd-main.cc:87] Couldn't open transport for hostname:11423 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))

I have used a customized service name for impala, can we do that? if not how can we achieve it?

principal - impala_<some text>@hostname@Domain

[#2]

@pandu2022 ,

When using Kerberos and/or TLS, please make sure that the hostname is specified as a fully qualified name (e.g. hostname.acm.com), instead of a short name.

Are you using a fully qualified name? If not, could you please try again using one?

Also, are you using a load balancer?

Regards,

André

[#3]

@araujo yes im using a load balancer as well

[#4]

@araujo 
yes I'm using fully qualified domain name
when the principal is like impala/<fqdn of host>@domain catalog server is able to connect to statestore successfully. but when the principal service name is custom as impala_test/<fqdn of host>@domain, statestore error log is updating as below,

I0321 08:30:30.615939 22113 statestore.cc:610] Creating new topic: ''catalog-update' on behalf of subscriber: 'catalog-server@<fqdn of catalog service host>:11426
I0321 08:30:30.615953 22113 statestore.cc:618] Registering: catalog-server@<fqdn of catalog service host>:11426
I0321 08:30:30.615984 22113 statestore.cc:641] Subscriber 'catalog-server@<fqdn of catalog service host>:11426' registered (registration id: c54a83a37fd90f6b:9023e9873ba17d89)
E0321 08:30:30.632500 21923 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
E0321 08:30:30.632500 21901 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:30.632710 21901 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:30.632715 21923 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:30.632727 21923 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:30.632732 21923 failure-detector.cc:91] 1 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is OK
I0321 08:30:30.632755 21901 statestore.cc:970] Unable to send topic update message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
E0321 08:30:31.651836 21924 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:31.651938 21924 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:31.651949 21924 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:31.651954 21924 failure-detector.cc:91] 2 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is OK
E0321 08:30:32.646282 21903 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:32.646412 21903 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:32.646428 21903 statestore.cc:970] Unable to send topic update message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
E0321 08:30:32.681665 21923 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:32.681779 21923 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:32.681805 21923 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:32.681810 21923 failure-detector.cc:91] 3 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is OK
E0321 08:30:33.697129 21926 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:33.697227 21926 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:33.697238 21926 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:33.697243 21926 failure-detector.cc:91] 4 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is OK
E0321 08:30:34.664945 21905 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:34.665043 21905 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:34.665056 21905 statestore.cc:970] Unable to send topic update message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
E0321 08:30:34.713243 21927 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:34.713331 21927 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:34.713342 21927 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:34.713347 21927 failure-detector.cc:91] 5 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is SUSPECTED
E0321 08:30:35.725081 21928 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)

[#5]

@pandu2022 ,

Where did you configure the customized service name for Impala?

Did you configure this since Impala was installed or was it initially using the default name and you later changed it?

Cheers,

André

[#6]

yes, initially i used a default service name as "impala".but later for a requirement i needed to use customize the service name part in principal as "impala_test". 
additionally, i tried including this customized service name in internal_principals_whitelist parameter as well but no good. 😥

@araujo thank you very much for replying. kudos

[#7]

Did you change this configuration in Cloudera Manager? Can you share screenshots of your configuration?

André

[#8]

We do not use Cloudera Manager to manage our impala cluster. It is a proprietary system. So I have limitations on sharing content here. Im sorry. We use start-up configs to start impala daemons to acquire expected behaviour .

Thanks,
Panduka

[#9]

@pandu2022 ,

Understood. Just keep in mind that not knowing any details makes it more difficult to help.

• What's your Kerberos KDC? (AD, MIT, FreeIPA, or other)
• When you changed the Impala principal name, did you create the new principal in Kerberos? You need to make sure that all the principals "impala_test/<host>" exist in the KDC for all the hosts.
• Did you regenerate the keytabs for all the Impala Daemons, Catalog and State Store with the new principal name?

Cheers,

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#10]

Hi André,
Please find the in line comments,

• What's your Kerberos KDC? (AD, MIT, FreeIPA, or other)
  [AWS Managed AD]
• When you changed the Impala principal name, did you create the new principal in Kerberos? You need to make sure that all the principals "impala_test/<host>" exist in the KDC for all the hosts.
  [yes i created principals with customized service name]
• Did you regenerate the keytabs for all the Impala Daemons, Catalog and State Store with the new principal name?
  [yes]

Thanks,
Panduka.