Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark installation 1.4 on hdp 2.3.4

avatar

How to install spark 1.4 on HDP 2.3.4. Because by default version on HDP 2.3.4 spark version is 1.5.

1 ACCEPTED SOLUTION

avatar
Super Guru

I think you can't do it through Ambari, either you can install it manually and build it from spark 1.4 source code. We recommend to keep one spark version which comes with HDP stack.

https://spark.apache.org/docs/1.4.1/building-spark.html

View solution in original post

10 REPLIES 10

avatar

you want to install manually or through ambari ?

avatar
Super Guru

I think you can't do it through Ambari, either you can install it manually and build it from spark 1.4 source code. We recommend to keep one spark version which comes with HDP stack.

https://spark.apache.org/docs/1.4.1/building-spark.html

avatar
Super Guru
@Ashnee Sharma

Instead of downgrading the spark version it would be better to work on the actual issue you are facing on spark 1.5 i.e java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericMutableRow cannot be cast to org.apache.spark.sql.Row

avatar
Super Guru

@Ashnee Sharma I can see one patch Phoenix patch in bug https://issues.apache.org/jira/browse/PHOENIX-2287, If you are suing Phoenix then you probably need to apply it in your phoenix env.

avatar

can you brif me how to apply it in phoenix env?

avatar
Super Guru

Sure, we will discuss this on this https://community.hortonworks.com/questions/39369/pheniox-patch-apply.html thread. Lets close this one.

avatar
Super Guru

Hi @Ashnee Sharma please accept this answer if you are good with the provided information to close this thread.

avatar

@jitendra

earlier we have HDP 2.3.2 on that spark version was 1.4 after upgrade the hdp verion to 2.3.4 the spark version also upgraded, Now we want to downgrade spark version to 1.4 from 1.5. Because we are getting issue to convert the java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericMutableRow cannot be cast to org.apache.spark.sql.Row

avatar

issue is with spark binary only.

link.:- https://issues.apache.org/jira/browse/PHOENIX-2287