Member since
01-14-2017
17
Posts
0
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
10717 | 05-19-2017 10:54 AM | |
16829 | 05-17-2017 03:05 PM | |
1746 | 05-15-2017 09:35 AM |
02-15-2018
08:56 AM
Hi, I have follwed all the steps to install Spark2 in cloudera vm but getting following error when I am trying to Host spark2 parcel. CDH (5.8 and higher) parcel required for SPARK2 (2.2.0.cloudera2-1.cdh5.12.0.p0.232957) is not available. I am using CDH version : 5.12 and using http://archive.cloudera.com/spark2/parcels/2.2.0.cloudera2/ link for parcel. Please help me with the configurtion.
... View more
10-27-2017
11:33 PM
Thanks, i find the repo in cloudera, but what i means, i can not find these csd in the deployed server.
... View more
05-19-2017
10:54 AM
For the next poor schlub that encouters this weird behavior, I figured out a workaround which also helps me to pinpoint the problem. It turns out that the problem was with SqlContext. I realized that my SparkContext could create and manipulate RDD's all day without problem. The SQlContext however would not allow me to work with dataframes without an error. I found that if I stopped my SparkContext, created a new one, and then created a new SqlContext from that, everything worked fine. This leads me to believe that there was something going on with the SparkContext that I was being passed from SparkMagic. I've updated to Spark2 now and I don't seem to be having any troubles that I have seen yet with the SparkSession so I doubt I will be digging more into this.
... View more
05-15-2017
09:35 AM
I figured out what was causing this. One of the repo sites that i had configured was not being let through my proxy. Once I opened up the proxy to that repo site, the error went away. --Willie
... View more
04-03-2017
02:23 PM
I tried this and while my job is still running, it looks like it has gotten farther than it has in the past. Thanks!
... View more
03-08-2017
02:55 PM
A bit more info.... (and this is cross-posted in project jupyter list) I think that messaging is getting screwed up between Pyspark and Livy. When the last cell is executed, I will see this on the client side. 2017-03-08 22:24:48,505 INFO EventsHandler InstanceId: 0e1c8fd2-047e-4337-b264-5b64ba74de5a,EventName: notebookStatementExecutionStart,Timestamp: 2017-03-08 22:24:48.504920,SessionGuid: 03d14478-6adc-4b 34-abef-b9b6fd400543,LivyKind: pyspark,SessionId: 8,StatementGuid: f1933b11-b767-4a18-b311-c48901ad8369 2017-03-08 22:24:48,788 DEBUG Command Status of statement 8 is running. 2017-03-08 22:24:50,920 DEBUG Command Status of statement 8 is running. ...and it never comes back. On the livy end, I see 17/03/08 17:26:26 INFO ContextLauncher: 17/03/08 17:26:26 INFO scheduler.DAGScheduler: ResultStage 17 (collect at <stdin>:5) finished in 1.521 s 17/03/08 17:26:26 INFO ContextLauncher: 17/03/08 17:26:26 INFO scheduler.DAGScheduler: Job 8 finished: collect at <stdin>:5, took 3.729078 s 17/03/08 17:26:27 DEBUG RpcDispatcher: [ClientProtocol] Registered outstanding rpc 230 (com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult). 17/03/08 17:26:27 DEBUG KryoMessageCodec: Encoded message of type com.cloudera.livy.rsc.rpc.Rpc$MessageHeader (6 bytes) 17/03/08 17:26:27 DEBUG KryoMessageCodec: Encoded message of type com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult (91 bytes) 17/03/08 17:26:27 DEBUG KryoMessageCodec: Decoded message of type com.cloudera.livy.rsc.rpc.Rpc$MessageHeader (6 bytes) 17/03/08 17:26:27 DEBUG KryoMessageCodec: Decoded message of type com.cloudera.livy.rsc.rpc.Rpc$NullMessage (2 bytes) 17/03/08 17:26:27 DEBUG RpcDispatcher: [ClientProtocol] Received RPC message: type=REPLY id=230 payload=com.cloudera.livy.rsc.rpc.Rpc$NullMessage 17/03/08 17:26:28 DEBUG RpcDispatcher: [ClientProtocol] Registered outstanding rpc 231 (com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult). ad infinitum So, with my limited knowledge, it looks to me that Livy thinks it has sent a result to a finished job, but pyspark hasn't received it. Anyone seen this before? Any thoughts?
... View more