Zeppelin reports connection refused with Spark

[#1]

%pyspark

java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at java.net.Socket.connect(Socket.java:556)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:661)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:622)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:540)
at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:706)
at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:686)
at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivyVersion(BaseLivyInterpreter.java:472)
at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:161)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[#2]

@PrernaU From the stack trace I can see you are using livy interpreter, Can you check is the livy url is set correctly in interpreter settings

[#3]

If I try to check the interpreter settings on Zeppelin UI I am getting below error , which shows that I don't have the required permissions I have logged in as admin user.

[#4]

@PrernaU  To access the interpreter page remove the below value from  Zeppelin Shiro Urls Block in zeppelin configuration or add the correct role name in place of zeppelin_admin_group

/api/interpreter/** = authc, roles[{{zeppelin_admin_group}}]

[#5]

@Scharan the livy url is : http://localhost:8998

When I tired to access the webui It was not responding.

So I restarted the live server ,

[root@cdp1pub spark]# export SPARK_HOME=/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/

[root@cdp1pub hadoop]# export HADOOP_CONF_DIR=/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/etc/hadoop

[root@cdp1pub hadoop]# /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/livy2/bin/livy-server start
starting java -cp /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/livy2/jars/*:/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/livy2/conf:/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/etc/hadoop: org.apache.livy.server.LivyServer, logging to /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/livy2/logs/livy-root-server.out
[root@cdp1pub hadoop]# /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/livy2/bin/livy-server status
livy-server is running (pid: 24354)
[root@cdp1pub hadoop]#

Now I am able to login and 

%pyspark is not throwing any error

Thanks a lot for help

[#6]

One more thing I only see livy , md and angular as the interpreter option , why am I not seeing python and sh options as well.

[#7 · Accepted solution]

why am I not seeing python and sh options as well?

# you need to install python and sh interpreter manually, refer to the following doc on how to install it

https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/installation.html

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.