Support Questions

vishwanath_voru · ‎07-18-2017

@Ali Bajwa or others can you help me on this...

I did install Solr manually and kerberized the same using Windows AD server.

Below is the link I used to integrate Solr with Kerberos.

https://community.hortonworks.com/articles/15159/securing-solr-collections-with-ranger-kerberos.html

When I try to start solr in cluster mode, I see an error which is like below:

**************************************************************

[root@nn solr]# service solr status

Found 1 Solr nodes: Solr process 1490 running on port 8983 INFO - 2017-07-18 08:05:06.411; org.apache.solr.util.SolrCLI; Set HttpClientConfigurer from: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer INFO - 2017-07-18 08:05:06.572; org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer; Setting up SPNego auth with config: /opt/solr/bin/jaas.conf Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /etc/security/keytabs/solr.service_nn.keytab refreshKrb5Config is false principal is solr/nn.wwtest.net@WWTEST.NET tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is solr/nn.wwtest.net@WWTEST.NET Will use keytab Commit Succeeded ERROR: Failed to get system information from http://nn.wwtest.net:8983/solr due to: org.apache.http.client.ClientProtocolException: Expected JSON response from server but received: <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> <title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - RC4 with HMAC)</title> </head> <body><h2>HTTP ERROR 403</h2> <p>Problem accessing /solr/admin/info/system. Reason: <pre> GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - RC4 with HMAC)</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/> </body> </html> Typically, this indicates a problem with the Solr server; check the Solr server logs for more information.

**************************************************************

[root@nn keytabs]# klist -kte solr.service_dn1.keytab Keytab name: FILE:solr.service_dn1.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 1 01/01/1970 00:00:00 solr/dn1.wwtest.net@WWTEST.NET (des-cbc-crc) 1 01/01/1970 00:00:00 solr/dn1.wwtest.net@WWTEST.NET (des-cbc-md5) 1 01/01/1970 00:00:00 solr/dn1.wwtest.net@WWTEST.NET (arcfour-hmac) 1 01/01/1970 00:00:00 solr/dn1.wwtest.net@WWTEST.NET (aes256-cts-hmac-sha1-96) 1 01/01/1970 00:00:00 solr/dn1.wwtest.net@WWTEST.NET (aes128-cts-hmac-sha1-96) [root@nn keytabs]#

**************************************************************

[root@nn keytabs]# kinit -kt /etc/security/keytabs/solr.service_dn1.keytab solr/dn1.wwtest.net@WWTEST.NET [root@nn keytabs]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: solr/dn1.wwtest.net@WWTEST.NET Valid starting Expires Service principal 07/18/2017 08:42:49 07/18/2017 18:42:49 krbtgt/WWTEST.NET@WWTEST.NET renew until 07/25/2017 08:42:49 [root@nn keytabs]#

**************************************************************

Can some one please help.

VR46 · ‎07-19-2017

Hello @Vishwanath Voruganti,

What is happening is this:

1. When Solr is starting up, it is trying to authenticate to AD using solr.service_nn.keytab.

2. During authentication, AD server responds back to Solr which is encrypted with RC4-HMAC encryption (default for AD)

3. Solr is trying to decrypt this response using keytab (which has key only for AES128 encryption).

4. Hence Solr is not able to get authenticated with AD and is giving the error -

Cannot find key of appropriate type to decrypt AP REP - RC4 with HMAC

The quickest way to solve this problem is to generate a keytab in AD for "sorl/nn.wwtest.net" with RC4-HMAC encryption and use the same on Solr node.

Hope this helps.

PS - I don't understand why you used "solr.service_nn.keytab" in configuration but listed "solr.service_dn1.keytab" in command output.

vishwanath_voru · ‎07-20-2017

@Vipin Rahor:

Thanks for your time.

The quickest way to solve this problem is to generate a keytab in AD for "sorl/nn.wwtest.net" with RC4-HMAC encryption and use the same on Solr node.

[ Vishwa ]: I did that in another trail, but it did not work.

PS - I don't understand why you used "solr.service_nn.keytab" in configuration but listed "solr.service_dn1.keytab" in command output.

[ Vishwa ]: I gave the sample output of a keytab, I should have given the klist of exact keytab.

Vipin:

Is this an issue because of encryption mechanism?

I am forced to terminate the instances and I could not replicate the same now.

Thanks a lot for your time.

VR46 · ‎07-20-2017

@Vishwanath Voruganti

Please share the output of the "klist -kte <new-keytab>" here, so that I can see the current key encryptions types. That is, if you still have the keytab lying around.

Yes, this is the issue because of encryption type mismatch.

vishwanath_voru · ‎07-21-2017

@Vipin Rathor... Many many thanks for your time on this.

Server is scrapped now and I remember that all encryption types are present for the keytab and it is visible in klist command.

Unfortunately, I don't have the cluster now.

And do you see any other root cause to this issue if all the encryption types are present for the keytab?

VR46 · ‎07-22-2017

@Vishwanath Voruganti,

If the RC4-HMAC encryption type was present in keytab then the other reason could be stale keytab. That should get resolved once you regenerate the keytab. Other than these two, I don't see any reasons to get this specific error:

Cannot find key of appropriate type to decrypt AP REP - RC4 with HMAC

Hope this helps!

voruvishwa · ‎07-25-2017

@ Vipin Rathor

I am facing a different kind of issue and could you please help me on this.

[root@XXXXXdn2 keytabs]# curl --negotiate -u : http://XXXXdn1.xxxxx.net:8983/solr/shub_test/query?q=*.* <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> <title>Error 401 org.apache.hadoop.security.authentication.util.SignerException: Invalid signature</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /solr/test/query. Reason: <pre> org.apache.hadoop.security.authentication.util.SignerException: Invalid signature</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/> </body> </html>

Cloudera Community

Support Questions

Unable to start Solr service in kerberized environment.