Support Questions
Find answers, ask questions, and share your expertise

Configuration zeppelin /livy/ solr in kerberized cluster

Configuration zeppelin /livy/ solr in kerberized cluster

New Contributor

Hi,

In a kerberized cluster, I create a zeppelin notebook trough livy in which I want to do a solr call (example below, using spark-solr --

https://github.com/lucidworks/spark-solr)

%livy
val options = Map("collection" -> "myCollection", "zkhost" -> "<servername1>:2181,<servername2>:2181,<servername3>:2181/solr", "query" ->"body:test")
val solrSearchResult = sparkContext.read.format("solr").options(options).load
val nb = solrSearchResult.count

Solr authentification required error while executing:

org.apache.solr.common.SolrException: Request [http://<servername>:8983/solr/myCollection/schema/fieldtypes?wt=json] failed due to: HTTP/1.1 401 Authentication required: <html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/><title>Error 401 Authentication required</title></head><body><h2>HTTP ERROR 401</h2><p>Problem accessing /solr/myCollection/schema/fieldtypes. Reason:<pre> Authentication required</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/></body></html>

If I execute the same script in the spark-shell, it works, if I add the below spark-submit options:

--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/home/myusername/client_jaas.conf" --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=/home/myusername/client_jaas.conf" --files "/home/myusername/client_jaas.conf,/etc/security/keytabs/myusername.keytab"

How/where can I add these spark-submit options for zeppelin/livy?

I tried export SPARK_SUBMIT_OPTIONS in zeppelin configuration, without success.

And where should be located the jaas conf file and keytab file? on hdfs?

Thanks in advance for your help

6 REPLIES 6

Re: Configuration zeppelin /livy/ solr in kerberized cluster

Expert Contributor

Try adding the options to livy env in dep

env LIVY_REPL_JAVA_OPTS="-Dhdp.version=2.3.2.0-2950" 

This should give you what you want.

For reference: https://github.com/cloudera/hue/commit/7289799f5c314aeae1b636bc32b0bcdf67def7f3

Re: Configuration zeppelin /livy/ solr in kerberized cluster

Expert Contributor

Actually, You could also probably just add them as config in your interpreter. I'm not sure you need do use the above parameter setting.

Re: Configuration zeppelin /livy/ solr in kerberized cluster

New Contributor

Thanks for your points:

- I tried LIVY_REPL_JAVA_OPTS, without success

- I tried to add in the livy interpreter: I think indeed that it's a good place:

livy.spark.executor.extraJavaOptions and livy.spark.executor.extraJavaOptions

But I still have the same error (Authentication required). But maybe this is an access issue: where should be located the jaas file? on hdfs? In this case, in the jaas file, I should refer to a keytab (also located on hdfs?). I don't know for the rights on these files : zeppelin, livy or spark user? This is not clear to me.

jaas conf file is in this case:

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="hdfs:///user/zeppelin/zeppelin.server.kerberos.keytab"
  storeKey=true
  useTicketCache=false
  doNotPrompt=true
  debug=true
  principal="zeppelin@MYREALM.COM" ;
};

And the jaas conf and zeppelin.server.kerberos.keytab are on hdfs:///user/zeppelin

In the interpreter:

livy.spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/user/zeppelin/client_jaas.conf
livy.spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/user/zeppelin/client_jaas.conf

cf. in the logs: although /tmp/client_jaas.conf exists (on hdfs)

 zookeeper.ClientCnxn: SASL configuration failed: javax.security.auth.login.LoginException: Zookeeper client cannot authenticate using the 'Client' section of the supplied JAAS configuration: '/tmp/client_jaas.conf' because of a RuntimeException: java.lang.SecurityException: java.io.IOException: /tmp/client_jaas.conf (No such file or directory) Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.

What do think about?

Thanks!

Re: Configuration zeppelin /livy/ solr in kerberized cluster

Mentor

@mbodin

Your keytab is inaccessible in hdfs as Kerberos expects it on the local file system so it should look like this

Client { 
  com.sun.security.auth.module.Krb5LoginModule required 
  useKeyTab=true 
  keyTab="/etc/security/keytabs/zeppelin.server.kerberos.keytab" 
  storeKey=true 
  useTicketCache=false 
  doNotPrompt=true 
  debug=true 
  principal="zeppelin@MYREALM.COM" ; 
};

Please follow this document How to configure zeppelin livy interpreter for secure HDP cluster that should resolve your issue let me know

HTH

Re: Configuration zeppelin /livy/ solr in kerberized cluster

New Contributor

Hi,

I've just copied the client jaas conf on all namenodes and now I've got this error:

2018-10-01 10:21:52,691 INFO  [pool-7-thread-1] cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2018-10-01 10:21:52,701 WARN  [pool-7-thread-1-SendThread(<servername>:2181)] zookeeper.ClientCnxn: SASL configuration failed: javax.security.auth.login.LoginException: Unable to obtain password from user
 Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.

I think the jaas file has been well taken into account (that's the good point)

If I do a kinit on the server with the principal and keytab defined in the jaas file (as zeepelin user), it works . So I don't really understand the "Unable to obtain password from user" error.

I'll continue to search, but if you have any idea, I'll be very happy to have some feedback.

Thanks,

Re: Configuration zeppelin /livy/ solr in kerberized cluster

New Contributor

okay, it works if the jaas conf is owned by the user connected into zeppelin and not by the zeppelin user.

So issue closed. Thanks