Zeppelin Spark interpreter on a kerberized cluster with encrypted zones fails

[#1]

We're using HDP 2.6.0.3 with Active Directory/kerberos and using Ranger/Ranger KMS to handle encrypted zones. If we try to get data from this encrypted zone via %spark2 interpreter in Zeppelin like

%spark2.sql
select * from encrypted_datalake.artikel_ref limit 30

, we've got the following error in the spark interpreter log:

Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User:zeppelin not allowed to do 'DECRYPT_EEK' on 'bi-master-key'

Maybe that's why the delecation user zeppelin has not the right to decrypt the key from the encrypted zone. But the user from my login has this right and the %jdbc interpreter that is using hive as delegation user has this access and I can query data from this zone like

%jdbc(hive)
select * from encrypted_datalake.artikel_ref limit 10

without any errors. How can switch the zeppelin user to a kerberized user?

[#2 · Accepted solution]

@Ramon Wartala

By design, zeppelin's spark and spark2 interpreters would always execute your query as 'zeppelin' user and they dont support user impersonation. Hence it is bound to fail if 'zeppelin' user doesn't have the permissions to decrypt the key.

jdbc, livy and livy2 interpreters support user impersonation and so your scenario would pass with any of these : %livy.sql, %livy2.sql and %jdbc(hive)

[#3]

Ok, I see. But why the livy and livy2 interpreter are not installed in HDP 2.6.0.3 per default? I can't find a installation routine for both interpreters.

[#4]

@Ramon Wartala

I would suggest to check if Livy and Livy2 are present under Spark and Spark2 services respectively . If Livy and Livy2 servers are not installed on the cluster, then corresponding interpreters wont be present in Zeppelin

check this out : https://issues.apache.org/jira/browse/AMBARI-19919

[#5]

@Kshitij Badani, you're right. I restart the Livy2 server and remove the Zeppelin service from Ambari and clean all config files on the host location and reinstall the Zeppelin service. After that, the Livy2 interpreter was available. But now, I've got an error if I try to connect with it. The zeppelin-interpreter-livy2-livy-zeppelin...log shows me the following error:

ERROR [2017-06-29 18:54:04,427] ({pool-2-thread-7} BaseLivyInterprereter.java[callRestAPI]:416) - Error with 401 StatusCode:
ERROR [2017-06-29 18:54:04,427] ({pool-2-thread-7} BaseLivyInterprereter.java[createSession]:214) - Error when creating livy session for user r00138
org.apache.zeppelin.livy.LivyException: Error with 401 StatusCode:
	at org.apache.zeppelin.livy.BaseLivyInterprereter.callRestAPI(BaseLivyInterprereter.java:448)
	at org.apache.zeppelin.livy.BaseLivyInterprereter.createSession(BaseLivyInterprereter.java:191)
	at org.apache.zeppelin.livy.BaseLivyInterprereter.initLivySession(BaseLivyInterprereter.java:98)
	at org.apache.zeppelin.livy.BaseLivyInterprereter.open(BaseLivyInterprereter.java:80)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:482)
	at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
	at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

User r00138 is my kerberos user. Should I need to set zeppelin.livy.principal oder zeppelin.livy.keytab with the zeppelin proxysuser? Actually the user and the keytab is empty in my configuration. Or should I setup my user credentials under 'Credentials'?

[#6]

@Ramon Wartala

Please check this article to see if you are missing any of these configs?

https://community.hortonworks.com/articles/80059/how-to-configure-zeppelin-livy-interpreter-for-sec....

[#7]

I modify all properties as in this article and I checked every property twice but I've still got

javax.security.auth.login.LoginException: Unable to obtain password from user

inside Zeppelin. And

INFO [2017-06-30 08:44:02,849] ({Thread-0} RemoteInterpreterServer.java[run]:95) - Starting remote interpreter server on port 15012
 INFO [2017-06-30 08:44:03,209] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivyPySparkInterpreter
 INFO [2017-06-30 08:44:03,231] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkInterpreter
 INFO [2017-06-30 08:44:03,234] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkSQLInterpreter
 INFO [2017-06-30 08:44:03,235] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivyPySpark3Interpreter
 INFO [2017-06-30 08:44:03,237] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkRInterpreter
 INFO [2017-06-30 08:44:03,270] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1498805043269 started by scheduler interpreter_1470680829
ERROR [2017-06-30 08:44:03,640] ({pool-2-thread-2} BaseLivyInterprereter.java[createSession]:214) - Error when creating livy session for user r00138
org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is javax.security.auth.login.LoginException: Unable to obtain password from user

inside the

/var/log/zeppelin/zeppelin-interpreter-livy2-livy-zeppelin-hdp-cluster-master3.log

[#8]

@Ramon Wartala Please paste screenshot of livy2 interpreter configs and also full /etc/livy2/conf/livy.conf file from your livy2 server host

[#9]

# Generated by Apache Ambari. Fri Jun 30 14:11:54 2017


livy.environment production
livy.impersonation.enabled true
livy.repl.enableHiveContext true
livy.server.access_control.enabled true
livy.server.access_control.users livy,zeppelin
livy.server.auth.kerberos.keytab /etc/security/keytabs/spnego.service.keytab
livy.server.auth.kerberos.principal HTTP/_HOST@TCHIBO.TCHIBOROOT.NET
livy.server.auth.type kerberos
livy.server.csrf_protection.enabled true
livy.server.launch.kerberos.keytab /etc/security/keytabs/livy2.service.keytab
livy.server.launch.kerberos.principal livy/_HOST@TCHIBO.TCHIBOROOT.NET
livy.server.port 8999
livy.server.recovery.mode recovery
livy.server.recovery.state-store filesystem
livy.server.recovery.state-store.url /livy2-recovery
livy.server.session.timeout 3600000
livy.spark.master yarn-cluster
livy.superusers zeppelin-datalake

[#10]

@Ramon Wartala Please attach screenshot of livy2 interpreter config as well. Also, Likewise in this article, https://discuss.pivotal.io/hc/en-us/articles/201914097-Hadoop-daemons-in-a-secured-cluster-fails-to-...

are you seeing any statement like this in your zeppelin logs?

java.io.IOException: Login failure for hdfs/dev6ha@SATURN.LOCAL from keytab /etc/security/phd/keytab/hdfs.service.keytab

[#11]

Hi @Kshitij Badani

got the same error as Ramon, so maybe my screens can help

I`ve got hdp 2.6.3, kerberized, using microsoft AD and want to impersonate users so thay can run spark 1/2 jobs. so far I`m trying to run livy with spark 1.6.3 but after logging in with AD user and running a note I`m getting

INFO [2018-05-10 16:49:41,905] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1525958424236_42692352 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1635594872
 INFO [2018-05-10 16:49:41,906] ({pool-2-thread-2} Paragraph.java[jobRun]:366) - run paragraph 20180510-152024_1120525270 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@5b439305
 INFO [2018-05-10 16:49:41,918] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:132) - Run interpreter process [/usr/hdp/current/zeppelin-server/bin/interpreter.sh, -d, /usr/hdp/current/zeppelin-server/interpreter/livy, -p,
 35361, -u, mvince, -l, /usr/hdp/current/zeppelin-server/local-repo/2CKX6DGQZ, -g, livy]
 INFO [2018-05-10 16:49:42,473] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivySparkInterpreter
 INFO [2018-05-10 16:49:42,963] ({pool-2-thread-2} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:578) - Push local angular object registry from ZeppelinServer to remote interpreter group 2CKX6DGQZ:mvince:
 INFO [2018-05-10 16:49:42,981] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivySparkSQLInterpreter
 INFO [2018-05-10 16:49:42,986] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivyPySparkInterpreter
 INFO [2018-05-10 16:49:42,992] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivyPySpark3Interpreter
 INFO [2018-05-10 16:49:42,997] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivySparkRInterpreter
 INFO [2018-05-10 16:49:43,005] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivySharedInterpreter
 WARN [2018-05-10 16:49:43,107] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2067) - Job 20180510-152024_1120525270 is finished, status: ERROR, exception: null, result: %text javax.security.auth.login.LoginException: Unable to obtain password from user

  at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
  at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
  at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
  at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
  at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
  at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
  at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
  at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:185)
  at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
  at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
  at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:619)
  at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:599)
  at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivyVersion(BaseLivyInterpreter.java:395)
  at org.apache.zeppelin.livy.LivySharedInterpreter.open(LivySharedInterpreter.java:47)
  at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
  at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivySharedInterpreter(BaseLivyInterpreter.java:165)
  at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:139)
  at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
  at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493)
  at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
  at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

any idea what I`m doing wrong?

[#12]

No, only if I try

%livy2.pyspark
print "1"

I've got the error

ERROR [2017-07-03 13:38:23,890] ({pool-2-thread-11} BaseLivyInterprereter.java[createSession]:214) - Error when creating livy session for user r00138
org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is javax.security.auth.login.LoginException: Unable to obtain password from user