Support Questions
Find answers, ask questions, and share your expertise

hue notebook pyspark

Super Collaborator

hi:

after type somenthing from pispark y have this error from livy server, but then livy server create another session, but with scala it work.

7859-snip20160922-2.png

01:25:51.007 [ForkJoinPool-1-worker-3] INFO  c.c.hue.livy.server.SessionManager - created session 2
xxxxxxxxx - - - 22/Sep/2016:01:25:51 +0200 "POST /sessions HTTP/1.1" 201 72
xxxxxxxxx - - - 22/Sep/2016:01:25:51 +0200 "GET /sessions/2 HTTP/1.1" 200 72
01:26:09.760 [qtp438135304-21] ERROR c.c.hue.livy.server.SessionServlet$ - internal error
java.util.concurrent.ExecutionException: dispatch.StatusCode: Unexpected response status: 500
 at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:342) ~[livy-assembly.jar:na]
 at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.abort(NettyAsyncHttpProvider.java:1418) ~[livy-assembly.jar:na]
 at com.ning.http.client.providers.netty.NettyAsyncHttpProvider$HttpProtocol.handle(NettyAsyncHttpProvider.java:2257) ~[livy-assembly.jar:na]
 at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.messageReceived(NettyAsyncHttpProvider.java:1228) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[livy-assembly.jar:na]
 at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[livy-assembly.jar:na]
 at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[livy-assembly.jar:na]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_65]
 at java.uti.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_65]
 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: dispatch.StatusCode: Unexpected response status: 500
 at dispatch.OkHandler$class.onStatusReceived(handlers.scala:37) ~[livy-assembly.jar:na]
 at dispatch.OkFunctionHandler.onStatusReceived(handlers.scala:29) ~[livy-assembly.jar:na]
 at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.updateStatusAndInterrupt(NettyAsyncHttpProvider.java:1560) ~[livy-assembly.jar:na]
 at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.access$2600(NettyAsyncHttpProvider.java:161) ~[livy-assembly.jar:na]
 at com.ning.http.client.providers.netty.NettyAsyncHttpProvider$HttpProtocol.handle(NettyAsyncHttpProvider.java:2212) ~[livy-assembly.jar:na]
 at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.messageReceived(NettyAsyncHttpProvider.java:1228) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[livy-assembly.jar:na]
 at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[livy-assembly.jar:na]
 at org.jboss.nety.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[livy-assembly.jar:na]
 at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[livy-assembly.jar:na]

 at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[livy-assembly.jar:na]

 ... 3 common frames omitted

xxxxxxxx - - - 22/Sep/2016:01:26:09 +0200 "GET /sessions/2/statements/0 HTTP/1.1" 500 93

Exception in thread "Thread-24" java.io.IOException: Stream closed

 at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)

 at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)

 at java.io.BufferedInputStream.read(BufferedInputStream.java:345)

 at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)

 at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)

 at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)

 at java.io.InputStreamReader.read(InputStreamReader.java:184)

 at java.io.BufferedReader.fill(BufferedReader.java:161)

 at java.io.BufferedReader.readLine(BufferedReader.java:324)

 at java.io.BufferedReader.readLine(BufferedReader.java:389)

 at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67)

 at scala.collection.Iterator$class.foreach(Iterator.scala:727)

 at scala.collection.AbstractIterator.foreach(Iterator.scala:1156)

 at com.cloudera.hue.livy.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:37)

16/09/22 01:26:21 INFO impl.YarnClientImpl: Killed application application_1474375921822_0069

16/09/22 01:26:21 INFO impl.YarnClientImpl: Killed application application_1474375921822_0069

xxxxxxxxx - - - 22/Sep/2016:01:26:21 +0200 "DELETE /sessions/2 HTTP/1.1" 200 17

01:26:21.065 [ForkJoinPool-1-worker-2] INFO  c.c.h.l.s.SparkSubmitProcessBuilder - Running spark-submit --master yarn-cluster --class com.cloudera.hue.livy.repl.Main --driver-java-options -Dlivy.repl.callback-url=http://xxxxxxxx:8998/sessions/3/callback -Dlivy.repl.port=0 --proxy-user hdfs /usr/local/hue/apps/spark/java-lib/livy-assembly.jar pyspark

01:26:21.067 [ForkJoinPool-1-worker-2] INFO  c.c.hue.livy.server.SessionManager - created session 3

xxxxxxxxx- - - 22/Sep/2016:01:26:21 +0200 "POST /sessions HTTP/1.1" 201 72

xxxxxxx - - - 22/Sep/2016:01:26:21 +0200 "GET /sessions/3 HTTP/1.1" 200 72


1 REPLY 1

Super Guru

have you tried the most recent version