Kerberos Authentication Failure : Catalog Server Unable to Connect to Statestore Port

[#1]

F0303 09:59:04.650674 32117 catalogd-main.cc:87] Couldn't open transport for hostname:11423 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))

I have used a customized service name for impala, can we do that? if not how can we achieve it?

principal - impala_<some text>@hostname@Domain

[#2]

@pandu2022 ,

When using Kerberos and/or TLS, please make sure that the hostname is specified as a fully qualified name (e.g. hostname.acm.com), instead of a short name.

Are you using a fully qualified name? If not, could you please try again using one?

Also, are you using a load balancer?

Regards,

André

[#3]

@araujo yes im using a load balancer as well

[#4]

@araujo 
yes I'm using fully qualified domain name
when the principal is like impala/<fqdn of host>@domain catalog server is able to connect to statestore successfully. but when the principal service name is custom as impala_test/<fqdn of host>@domain, statestore error log is updating as below,

I0321 08:30:30.615939 22113 statestore.cc:610] Creating new topic: ''catalog-update' on behalf of subscriber: 'catalog-server@<fqdn of catalog service host>:11426
I0321 08:30:30.615953 22113 statestore.cc:618] Registering: catalog-server@<fqdn of catalog service host>:11426
I0321 08:30:30.615984 22113 statestore.cc:641] Subscriber 'catalog-server@<fqdn of catalog service host>:11426' registered (registration id: c54a83a37fd90f6b:9023e9873ba17d89)
E0321 08:30:30.632500 21923 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
E0321 08:30:30.632500 21901 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:30.632710 21901 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:30.632715 21923 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:30.632727 21923 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:30.632732 21923 failure-detector.cc:91] 1 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is OK
I0321 08:30:30.632755 21901 statestore.cc:970] Unable to send topic update message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
E0321 08:30:31.651836 21924 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:31.651938 21924 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:31.651949 21924 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:31.651954 21924 failure-detector.cc:91] 2 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is OK
E0321 08:30:32.646282 21903 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:32.646412 21903 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:32.646428 21903 statestore.cc:970] Unable to send topic update message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
E0321 08:30:32.681665 21923 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:32.681779 21923 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:32.681805 21923 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:32.681810 21923 failure-detector.cc:91] 3 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is OK
E0321 08:30:33.697129 21926 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:33.697227 21926 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:33.697238 21926 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:33.697243 21926 failure-detector.cc:91] 4 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is OK
E0321 08:30:34.664945 21905 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:34.665043 21905 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:34.665056 21905 statestore.cc:970] Unable to send topic update message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
E0321 08:30:34.713243 21927 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
I0321 08:30:34.713331 21927 thrift-client.cc:94] Unable to connect to <fqdn of catalog service host>:11434
I0321 08:30:34.713342 21927 statestore.cc:970] Unable to send heartbeat message to subscriber catalog-server@<fqdn of catalog service host>:11426, received error: Couldn't open transport for <fqdn of catalog service host>:11434 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database))
I0321 08:30:34.713347 21927 failure-detector.cc:91] 5 consecutive heartbeats failed for 'catalog-server@<fqdn of catalog service host>:11426'. State is SUSPECTED
E0321 08:30:35.725081 21928 authentication.cc:177] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)

[#5]

@pandu2022 ,

Where did you configure the customized service name for Impala?

Did you configure this since Impala was installed or was it initially using the default name and you later changed it?

Cheers,

André

[#6]

yes, initially i used a default service name as "impala".but later for a requirement i needed to use customize the service name part in principal as "impala_test". 
additionally, i tried including this customized service name in internal_principals_whitelist parameter as well but no good. 😥

@araujo thank you very much for replying. kudos

[#7]

Did you change this configuration in Cloudera Manager? Can you share screenshots of your configuration?

André

[#8]

We do not use Cloudera Manager to manage our impala cluster. It is a proprietary system. So I have limitations on sharing content here. Im sorry. We use start-up configs to start impala daemons to acquire expected behaviour .

Thanks,
Panduka

[#9]

@pandu2022 ,

Understood. Just keep in mind that not knowing any details makes it more difficult to help.

• What's your Kerberos KDC? (AD, MIT, FreeIPA, or other)
• When you changed the Impala principal name, did you create the new principal in Kerberos? You need to make sure that all the principals "impala_test/<host>" exist in the KDC for all the hosts.
• Did you regenerate the keytabs for all the Impala Daemons, Catalog and State Store with the new principal name?

Cheers,

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#10]

Hi André,
Please find the in line comments,

• What's your Kerberos KDC? (AD, MIT, FreeIPA, or other)
  [AWS Managed AD]
• When you changed the Impala principal name, did you create the new principal in Kerberos? You need to make sure that all the principals "impala_test/<host>" exist in the KDC for all the hosts.
  [yes i created principals with customized service name]
• Did you regenerate the keytabs for all the Impala Daemons, Catalog and State Store with the new principal name?
  [yes]

Thanks,
Panduka.

[#11]

@pandu2022 ,

Have you updated the properties below for *all* the Impala service roles (ID, catalog and statestore)?

• principal: When using a load-balancer this should be of the form impala_test/<LB_fqdn>@<REALM>. If not using a LB, this should be impala_test/<host_fqdn>@<REALM>.
• be_principal: This is only necessary when a LB is being used and should be of the form impala_test/<host_fqdn>@<REALM>.

Cheers,

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#12]

Hi,
everything works fine for "impala" service name. but I will try and let you know. Thank you very much.

Thanks,
Panduka.

[#13]

@araujo 
i do not know where else to configure this customized service name for impala services.

[#14]

@pandu2022 

I have feeling that your kerberos principle doesn't exist on the KDC server. On the statestore server can you try running "kinit impala_test/<host_fqdn>@<REALM>". If you get prompt for password that indicate your principle is exist on KDC server. If you get error (not found in kerberos database) when you kinit, that indicate your principle doesn't exist on the KDC server.

If the kinit works from catalog server, then most likely on statestore you are using different KDC server. In this case may be you should check your /etc/krb5.conf to make sure there are match.

rgds,

Ram.

[#15]

Hi Ram,
kinit works fine in both of the servers as expected and no difference in krb5.conf files as well.

Thanks,
Panduka.

[#16]

@araujo / @ram76 

should the backend host fqdn be accessible from the KDC server. I am using a custom fqdn for hosts which are accessible within the cluster but KDC can access only the LB fqdn.is this an issue? any comments?

[#17]

@pandu2022 ,

The KDC does not need to connect to Impala servers.

Do you happen to have multiple realms in your environment with cross-realm trust configured between them?

Could you please run the below commands and share the output?

kinit <your_user>
kvno impala/<host_fqdn>@<REALM>
kvno impala_test/<host_fqdn>@<REALM>

Cheers,

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#18]

@araujo 

OMG!!

kinit <user> - works fine

kvno impala/<host fqdn> - works fine

but,

kvno impala_test/<host fqdn> - not working !!!
kvno: Server not found in Kerberos database while getting credentials for impala_test/<host fqdn>@domain 

[#19]

and additionally ,
what does this kvno command do?
when i kinit relavant keytab for impala_test/<host fqdn>@domain. it works fine. but kvno command does not.

[#20]

@pandu2022 ,

Please check the servicePrincipalName (SPN) property of the AD user. It should be impala_test/<host>@realm.

André

[#21]

How did you create the impala_test principal?