Member since
04-20-2017
13
Posts
3
Kudos Received
0
Solutions
07-03-2017
12:00 PM
No, only if I try %livy2.pyspark
print "1"
I've got the error ERROR [2017-07-03 13:38:23,890] ({pool-2-thread-11} BaseLivyInterprereter.java[createSession]:214) - Error when creating livy session for user r00138
org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is javax.security.auth.login.LoginException: Unable to obtain password from user
... View more
06-30-2017
04:40 PM
# Generated by Apache Ambari. Fri Jun 30 14:11:54 2017
livy.environment production
livy.impersonation.enabled true
livy.repl.enableHiveContext true
livy.server.access_control.enabled true
livy.server.access_control.users livy,zeppelin
livy.server.auth.kerberos.keytab /etc/security/keytabs/spnego.service.keytab
livy.server.auth.kerberos.principal HTTP/_HOST@TCHIBO.TCHIBOROOT.NET
livy.server.auth.type kerberos
livy.server.csrf_protection.enabled true
livy.server.launch.kerberos.keytab /etc/security/keytabs/livy2.service.keytab
livy.server.launch.kerberos.principal livy/_HOST@TCHIBO.TCHIBOROOT.NET
livy.server.port 8999
livy.server.recovery.mode recovery
livy.server.recovery.state-store filesystem
livy.server.recovery.state-store.url /livy2-recovery
livy.server.session.timeout 3600000
livy.spark.master yarn-cluster
livy.superusers zeppelin-datalake
... View more
06-30-2017
03:37 PM
I modify all properties as in this article and I checked every property twice but I've still got javax.security.auth.login.LoginException: Unable to obtain password from user inside Zeppelin. And INFO [2017-06-30 08:44:02,849] ({Thread-0} RemoteInterpreterServer.java[run]:95) - Starting remote interpreter server on port 15012
INFO [2017-06-30 08:44:03,209] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivyPySparkInterpreter
INFO [2017-06-30 08:44:03,231] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkInterpreter
INFO [2017-06-30 08:44:03,234] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkSQLInterpreter
INFO [2017-06-30 08:44:03,235] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivyPySpark3Interpreter
INFO [2017-06-30 08:44:03,237] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkRInterpreter
INFO [2017-06-30 08:44:03,270] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1498805043269 started by scheduler interpreter_1470680829
ERROR [2017-06-30 08:44:03,640] ({pool-2-thread-2} BaseLivyInterprereter.java[createSession]:214) - Error when creating livy session for user r00138
org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is javax.security.auth.login.LoginException: Unable to obtain password from user
inside the /var/log/zeppelin/zeppelin-interpreter-livy2-livy-zeppelin-hdp-cluster-master3.log
... View more
06-29-2017
05:00 PM
1 Kudo
@Kshitij Badani, you're right. I restart the Livy2 server and remove the Zeppelin service from Ambari and clean all config files on the host location and reinstall the Zeppelin service. After that, the Livy2 interpreter was available. But now, I've got an error if I try to connect with it. The zeppelin-interpreter-livy2-livy-zeppelin...log shows me the following error: ERROR [2017-06-29 18:54:04,427] ({pool-2-thread-7} BaseLivyInterprereter.java[callRestAPI]:416) - Error with 401 StatusCode:
ERROR [2017-06-29 18:54:04,427] ({pool-2-thread-7} BaseLivyInterprereter.java[createSession]:214) - Error when creating livy session for user r00138
org.apache.zeppelin.livy.LivyException: Error with 401 StatusCode:
at org.apache.zeppelin.livy.BaseLivyInterprereter.callRestAPI(BaseLivyInterprereter.java:448)
at org.apache.zeppelin.livy.BaseLivyInterprereter.createSession(BaseLivyInterprereter.java:191)
at org.apache.zeppelin.livy.BaseLivyInterprereter.initLivySession(BaseLivyInterprereter.java:98)
at org.apache.zeppelin.livy.BaseLivyInterprereter.open(BaseLivyInterprereter.java:80)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:482)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
User r00138 is my kerberos user. Should I need to set zeppelin.livy.principal oder zeppelin.livy.keytab with the zeppelin proxysuser? Actually the user and the keytab is empty in my configuration. Or should I setup my user credentials under 'Credentials'?
... View more
06-28-2017
08:25 AM
Ok, I see. But why the livy and livy2 interpreter are not installed in HDP 2.6.0.3 per default? I can't find a installation routine for both interpreters.
... View more
06-27-2017
04:04 PM
2 Kudos
We're using HDP 2.6.0.3 with Active Directory/kerberos and using Ranger/Ranger KMS to handle encrypted zones. If we try to get data from this encrypted zone via %spark2 interpreter in Zeppelin like %spark2.sql
select * from encrypted_datalake.artikel_ref limit 30
, we've got the following error in the spark interpreter log: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User:zeppelin not allowed to do 'DECRYPT_EEK' on 'bi-master-key'
Maybe that's why the delecation user zeppelin has not the right to decrypt the key from the encrypted zone. But the user from my login has this right and the %jdbc interpreter that is using hive as delegation user has this access and I can query data from this zone like %jdbc(hive)
select * from encrypted_datalake.artikel_ref limit 10
without any errors. How can switch the zeppelin user to a kerberized user?
... View more
Labels:
05-30-2017
04:54 PM
We're using Microsoft Active Directory as KDC. I'm not sure how to check. But if I did the following: $ sudo su - zookeeper
$ kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/hdp-cluster-master1.apollon.mydomain.com@MYDOMAIN.MYDOMAINROOT.NET
$ klist
Ticket cache: FILE:/tmp/krb5cc_1002
Default principal: zookeeper/hdp-cluster-master1.apollon.mydomain.com@MYDOMAIN.MYDOMAINROOT.NET
Valid starting Expires Service principal
05/30/2017 18:52:38 05/31/2017 04:52:38 krbtgt/MYDOMAIN.MYDOMAINROOT.NET@MYDOMAIN.MYDOMAINROOT.NET
renew until 06/06/2017 18:52:38
... View more
05-30-2017
04:28 PM
The keytab generation working fine: 30 May 2017 18:26:08,584 INFO [Server Action Executor Worker 687]
CreateKeytabFilesServerAction:193 - Creating keytab file for
HTTP/hdp-cluster-master2.apollon.mydomain.com@MYDOMAIN.MYDOMAINROOT.NET
on host hdp-cluster-master2.apollon.mydomain.com
30 May 2017 18:26:08,639 INFO [Server Action Executor Worker 687]
CreateKeytabFilesServerAction:193 - Creating keytab file for
ambari-qa-datalake@MYDOMAIN.MYDOMAINROOT.NET on host
hdp-cluster-master2.apollon.mydomain.com
30 May 2017 18:26:08,678 INFO [Server Action Executor Worker 687]
CreateKeytabFilesServerAction:193 - Creating keytab file for
nn/hdp-cluster-master2.apollon.mydomain.com@MYDOMAIN.MYDOMAINROOT.NET on
host hdp-cluster-master2.apollon.mydomain.com
30 May 2017 18:26:08,711 INFO [Server Action Executor Worker 687]
CreateKeytabFilesServerAction:193 - Creating keytab file for
hdfs-datalake@MYDOMAIN.MYDOMAINROOT.NET on host
hdp-cluster-master2.apollon.mydomain.com
30 May 2017 18:26:08,751 INFO [Server Action Executor Worker 687]
CreateKeytabFilesServerAction:193 - Creating keytab file for
jhs/hdp-cluster-master2.apollon.mydomain.com@MYDOMAIN.MYDOMAINROOT.NET
on host hdp-cluster-master2.apollon.mydomain.com
... View more
05-30-2017
04:23 PM
I can confirm that I've got the right FQDNs on all of my worker and master nodes $ hostname -f
hdp-cluster-master1.apollon.mydomain.com
... View more
05-30-2017
01:30 PM
After we kerberize our cluster, the Zookeep is the only service that starts again. But if we check the Zookeeper service from Ambari with the Service check, the check fails like this: 2017-05-30 15:01:15,780 - File['/var/lib/ambari-agent/tmp/zkSmoke.out'] {'action': ['delete']}
2017-05-30 15:01:15,781 - File['/var/lib/ambari-agent/tmp/zkSmoke.sh'] {'content': StaticFile('zkSmoke.sh'), 'mode': 0755}
2017-05-30 15:01:15,782 - Execute['/var/lib/ambari-agent/tmp/zkSmoke.sh /usr/hdp/current/zookeeper-client/bin/zkCli.sh ambari-qa /usr/hdp/current/zookeeper-client/conf 2181 True /usr/bin/kinit /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-datalake@MYDOMAIN.MYDOMAINROOT.NET /var/lib/ambari-agent/tmp/zkSmoke.out'] {'logoutput': True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'try_sleep': 5}
zk_node1=hdp-cluster-master1.apollon.mydomain.com
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /zk_smoketest
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:703)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:363)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /zk_smoketest
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:698)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:363)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
Running test on host hdp-cluster-master1.apollon.mydomain.com
Connecting to hdp-cluster-master1.apollon.mydomain.com:2181
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Welcome to ZooKeeper!
JLine support is enabled
[zk: hdp-cluster-master1.apollon.mydomain.com:2181(CONNECTING) 0] get /zk_smoketest
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
... View more
Labels:
- Labels:
-
Apache Ambari