About srowen

srowen · ‎07-26-2015

Yes that shows the problem directly. You function has a reference to the instance of the outer class cc, and that is not serializable. You'll probably have to locate how your function is using the outer class and remove that. Or else the outer class cc has to be serializable.

srowen · ‎07-25-2015

You would really have to show more of the error, like, what is not serializable? typically that will point you to the problem. Something in the closure of your function is being dragged along when it's serialized, and it is not serializable.

srowen · ‎07-21-2015

Ah, I think I'm mistaken. Try this; note the capitalization: import sqlContext.implicits._

srowen · ‎07-21-2015

I meant in the import; you're missing the implicits, I think. import org.apache.spark.sql.SQLContext.implicits._

srowen · ‎07-21-2015

I think you're missing the package name. org.apache.spark.sql.SQLContext...

srowen · ‎07-10-2015

It's already complete and in 1.4. This was the initial JIRA https://issues.apache.org/jira/browse/SPARK-4924

srowen · ‎07-06-2015

Yes that could also be a cause. Is it possible to run the process inside the firewall? certainly the MapReduce jobs are intended to be managed by the Computation Layer from within the cluster.

srowen · ‎07-03-2015

If it were me, I'd download the source for 1.4.0 and build for the exact CDH artifacts to be safest. See http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html Then just try running the local copy of bin/spark-shell etc from that distribution. You need to use YARN masters. I won't 100% guarantee that works but see no reason it wouldn't. The build flags are probably like -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.3 -Pyarn

srowen · ‎07-03-2015

PS I should say too that you should be able to use 1.4 with CDH 5.4 and have it generally work; this requires a little bit of understanding of how to get a build on a machine and run from that build, but otherwise it's a YARN app and modulo some dependency issues at the edge maybe, should just work.

srowen · ‎07-03-2015

Presumably CDH 5.5, since a new minor release is needed to update a minor release of a component in general. There aren't timeframes for this, but you can see CDH is typically on a 4-6 month minor release cycle and 5.4 was out 2 months ago.

Online	Offline
Last Visited	‎02-13-2018 12:34 PM

Member Since	‎08-11-2014 09:17 AM
Last Visited	‎02-13-2018 12:34 PM
Posts	481
Kudos received	87

Cloudera Community

Re: Own code editor in CDSW?

Re: error using Pandas within PySpark transformati...

Re: Does CDSW need to be part of the cluster?

Re: Local Data combined with HDFS

Re: Where can I find Oryx 1.x releases (or GitHub)

Re: Spark Exception: Task Not Serializable

Re: Spark Exception: Task Not Serializable

Re: Spark/Scala Error: value toDF is not a member ...

Re: Spark/Scala Error: value toDF is not a member ...

Re: Spark/Scala Error: value toDF is not a member ...

Re: What dependencies to submit Spark jobs program...

Re: Run Oryx on a machine that is not part of the ...

Re: Which CDH release will include Spark 1.4.x?

Re: Which CDH release will include Spark 1.4.x?

Re: Which CDH release will include Spark 1.4.x?