Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark backward compatibility 1.6 vs 2.0

spark backward compatibility 1.6 vs 2.0

New Contributor

I am using spark 1.6 API in my application and now we plan to move to spark 2.0, and when I check the documentation below, there are only a few features are backward compatible, does that mean I have change most of my code ? , please advice.

One of the largest changes in Spark 2.0 is the new updated APIs:

Unifying DataFrame and Dataset: In Scala and Java, DataFrame and Dataset have been unified, i.e. DataFrame is just a type alias for Dataset of Row. In Python and R, given the lack of type safety, DataFrame is the main programming interface. SparkSession: new entry point that replaces the old SQLContext and HiveContext for DataFrame and Dataset APIs. SQLContext and HiveContext are kept for backward compatibility. A new, streamlined configuration API for SparkSession Simpler, more performant accumulator API A new, improved Aggregator API for typed aggregation in Datasets

thanks Pradeep

3 REPLIES 3

Re: spark backward compatibility 1.6 vs 2.0

Contributor

@pradeep arumalla SqlContext and HiveContext are not going away. Spark is big on backward compatibility. So these API’s are still available. Though from documentation it’s clear that they want developers to use SparkSession going forward.

Re: spark backward compatibility 1.6 vs 2.0

Expert Contributor

If you intend to adopt 2.0 then the recommendation is to use the new 2.0 APIs since the 1.0 API may either be deprecated or not have new features supported via them.

Highlighted

Re: spark backward compatibility 1.6 vs 2.0

Expert Contributor

If you intend to adopt 2.0 then the recommendation is to use the new 2.0 APIs since the 1.0 API may either be deprecated or not have new features supported via them.

Don't have an account?
Coming from Hortonworks? Activate your account here