Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Zeppelin Spark interpreter on a kerberized cluster with encrypted zones fails

avatar
Contributor

We're using HDP 2.6.0.3 with Active Directory/kerberos and using Ranger/Ranger KMS to handle encrypted zones. If we try to get data from this encrypted zone via %spark2 interpreter in Zeppelin like

%spark2.sql
select * from encrypted_datalake.artikel_ref limit 30

, we've got the following error in the spark interpreter log:

Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User:zeppelin not allowed to do 'DECRYPT_EEK' on 'bi-master-key'

Maybe that's why the delecation user zeppelin has not the right to decrypt the key from the encrypted zone. But the user from my login has this right and the %jdbc interpreter that is using hive as delegation user has this access and I can query data from this zone like

%jdbc(hive)
select * from encrypted_datalake.artikel_ref limit 10

without any errors. How can switch the zeppelin user to a kerberized user?

1 ACCEPTED SOLUTION

avatar

@Ramon Wartala

By design, zeppelin's spark and spark2 interpreters would always execute your query as 'zeppelin' user and they dont support user impersonation. Hence it is bound to fail if 'zeppelin' user doesn't have the permissions to decrypt the key.

jdbc, livy and livy2 interpreters support user impersonation and so your scenario would pass with any of these : %livy.sql, %livy2.sql and %jdbc(hive)

View solution in original post

11 REPLIES 11

avatar
Explorer

Hi @Kshitij Badani

got the same error as Ramon, so maybe my screens can help

I`ve got hdp 2.6.3, kerberized, using microsoft AD and want to impersonate users so thay can run spark 1/2 jobs. so far I`m trying to run livy with spark 1.6.3 but after logging in with AD user and running a note I`m getting

INFO [2018-05-10 16:49:41,905] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1525958424236_42692352 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1635594872
INFO [2018-05-10 16:49:41,906] ({pool-2-thread-2} Paragraph.java[jobRun]:366) - run paragraph 20180510-152024_1120525270 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@5b439305
INFO [2018-05-10 16:49:41,918] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:132) - Run interpreter process [/usr/hdp/current/zeppelin-server/bin/interpreter.sh, -d, /usr/hdp/current/zeppelin-server/interpreter/livy, -p,
35361, -u, mvince, -l, /usr/hdp/current/zeppelin-server/local-repo/2CKX6DGQZ, -g, livy]
INFO [2018-05-10 16:49:42,473] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivySparkInterpreter
INFO [2018-05-10 16:49:42,963] ({pool-2-thread-2} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:578) - Push local angular object registry from ZeppelinServer to remote interpreter group 2CKX6DGQZ:mvince:
INFO [2018-05-10 16:49:42,981] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivySparkSQLInterpreter
INFO [2018-05-10 16:49:42,986] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivyPySparkInterpreter
INFO [2018-05-10 16:49:42,992] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivyPySpark3Interpreter
INFO [2018-05-10 16:49:42,997] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivySparkRInterpreter
INFO [2018-05-10 16:49:43,005] ({pool-2-thread-2} RemoteInterpreter.java[init]:246) - Create remote interpreter org.apache.zeppelin.livy.LivySharedInterpreter
WARN [2018-05-10 16:49:43,107] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2067) - Job 20180510-152024_1120525270 is finished, status: ERROR, exception: null, result: %text javax.security.auth.login.LoginException: Unable to obtain password from user

at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:185)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:619)
at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:599)
at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivyVersion(BaseLivyInterpreter.java:395)
at org.apache.zeppelin.livy.LivySharedInterpreter.open(LivySharedInterpreter.java:47)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivySharedInterpreter(BaseLivyInterpreter.java:165)
at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:139)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

any idea what I`m doing wrong?


selection-110.pngselection-105.pngselection-104.pngselection-109.png

avatar
Contributor

No, only if I try

%livy2.pyspark
print "1"

I've got the error

ERROR [2017-07-03 13:38:23,890] ({pool-2-thread-11} BaseLivyInterprereter.java[createSession]:214) - Error when creating livy session for user r00138
org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is javax.security.auth.login.LoginException: Unable to obtain password from user