Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Zeppelin User Impersonation with Spark2 Interpreter is failing with Null Pointer Exception

Zeppelin User Impersonation with Spark2 Interpreter is failing with Null Pointer Exception

Expert Contributor

What am I doing:

1) I'm integrating spark with YARN ==> In this I.m using YARN NodeLabels (only on one node) and also making SPARK to launch ApplicationMaster(AM)'s only on the NodeLabel node using the parameter spark.yarn.am.nodeLabelExpression

2) Integrate cluster and zeppelin with AD ==> No Kerberos implemented

3) Now i'm enabling the Zeppelin User Impersonation with Per User and Isolated Per User zeppelin-spark2-interpreter.jpg

4) With this it should be able to submit the Spark jobs from Zeppelin to YARN with user who submitted the job instead of Zeppelin user

It works as expected In DEV Cluster with HDP 2.6.0 and Zeppelin 0.7.0:

I have followed the steps given in What am I doing: for this cluster and it works i.e. if a job is submitted it runs as a END USER instead of Zeppelin user zeppelin-dev-test.jpg zeppelin-yarn-dev-test.jpg

It doesn't work in PROD Cluster with HDP 2.6.2 Zeppelin 0.7.2:

The same process is followed in the PROD cluster it is not working, to rule the zeppelin SSH permission issues as given in here

a) i have added zeppelin user to /etc/sudoers with NOPASSWD on all nodes of the cluster

b) Password less ssh is enabled between the zeppelin user and the END user on all the nodes of the cluster ==> Which is not required to setup export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c' and also export ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER=${ZEPPELIN_IMPERSONATE_USER} in zeppelin-env.sh

as of Zeppelin 0.7.0, but added this to rule out the possibility of hitting the issue

Please note these settings are not done in DEV cluster but it is still working where as PROD is failing with the error:

from zeppelin-zeppelin-hdpedge1.awsaap.corecompete.com.log

DEBUG [2017-10-26 11:25:31,230] ({Thread-29} ExtensionStack.java[process]:384) - Entering IDLE WARN [2017-10-26 11:25:31,380] ({pool-2-thread-8} NotebookServer.java[afterStatusChange]:2058) - Job 20171025-144607_168916897 is finished, status: ERROR, exception: null, result: %text java.lang.NullPointerException at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38) at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33) at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:385) at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:374) at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:819) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.spark.PySparkInterpreter.getSparkInterpreter(PySparkInterpreter.java:566) at org.apache.zeppelin.spark.PySparkInterpreter.createGatewayServerAndStartScript(PySparkInterpreter.java:209) at org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:163) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

From zeppelin-zeppelin-hdpedge1.awsaap.corecompete.com.out

INFO: Scanning for root resource and provider classes in the packages: org.apache.zeppelin.rest Oct 26, 2017 11:22:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses INFO: Root resource classes found: class org.apache.zeppelin.rest.InterpreterRestApi class org.apache.zeppelin.rest.NotebookRepoRestApi class org.apache.zeppelin.rest.ConfigurationsRestApi class org.apache.zeppelin.rest.LoginRestApi class org.apache.zeppelin.rest.NotebookRestApi class org.apache.zeppelin.rest.CredentialRestApi class org.apache.zeppelin.rest.ZeppelinRestApi class org.apache.zeppelin.rest.SecurityRestApi class org.apache.zeppelin.rest.HeliumRestApi Oct 26, 2017 11:22:56 AM com.sun.jersey.api.core.ScanningResourceConfig init INFO: No provider classes found. Oct 26, 2017 11:22:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.13 06/29/2012 05:14 PM' Oct 26, 2017 11:22:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages WARNING: The following warnings have been detected with resource and/or provider classes: WARNING: A sub-resource method, public javax.ws.rs.core.Response org.apache.zeppelin.rest.NotebookRestApi.getNoteList() throws java.io.IOException, with URI template, "/", is treated as a resource method WARNING: A sub-resource method, public javax.ws.rs.core.Response org.apache.zeppelin.rest.NotebookRestApi.createNote(java.lang.String) throws java.io.IOException, with URI template, "/", is treated as a resource method WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.zeppelin.rest.CredentialRestApi.getCredentials(java.lang.String) throws java.io.IOException,java.lang.IllegalArgumentException, should not consume any entity. WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.zeppelin.rest.InterpreterRestApi.listInterpreter(java.lang.String), should not consume any entity.

Both the clusters (DEV and PROD) are on AWS EC2 instances.

It looks like ZEPPELIN-2890, and not able to get what did i miss in PROD which i have done in DEV.

Please help me to come across this issue.

Thanks

Venkat

1 REPLY 1
Highlighted

Re: Zeppelin User Impersonation with Spark2 Interpreter is failing with Null Pointer Exception

Expert Contributor

Found a way to workaround this, more details in relation to this are given in: https://github.com/sudheer0553/zeppelin-user-impersonation/tree/master

Don't have an account?
Coming from Hortonworks? Activate your account here