question Re: spark installation 1.4 on hdp 2.3.4 in Archives of Support Questions (Read Only)

spark installation 1.4 on hdp 2.3.4

ashneesharma88 — Mon, 13 Jun 2016 16:34:24 GMT

How to install spark 1.4 on HDP 2.3.4. Because by default version on HDP 2.3.4 spark version is 1.5.

Re: spark installation 1.4 on hdp 2.3.4

dsharma — Mon, 13 Jun 2016 17:06:43 GMT

you want to install manually or through ambari ?

Re: spark installation 1.4 on hdp 2.3.4

jyadav — Mon, 13 Jun 2016 17:18:42 GMT

I think you can't do it through Ambari, either you can install it manually and build it from spark 1.4 source code. We recommend to keep one spark version which comes with HDP stack.

https://spark.apache.org/docs/1.4.1/building-spark.html

Re: spark installation 1.4 on hdp 2.3.4

ashneesharma88 — Mon, 13 Jun 2016 17:44:55 GMT

@jitendra

earlier we have HDP 2.3.2 on that spark version was 1.4 after upgrade the hdp verion to 2.3.4 the spark version also upgraded, Now we want to downgrade spark version to 1.4 from 1.5. Because we are getting issue to convert the java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericMutableRow cannot be cast to org.apache.spark.sql.Row

Re: spark installation 1.4 on hdp 2.3.4

jyadav — Mon, 13 Jun 2016 17:53:02 GMT

@Ashnee Sharma

Instead of downgrading the spark version it would be better to work on the actual issue you are facing on spark 1.5 i.e java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericMutableRow cannot be cast to org.apache.spark.sql.Row

Re: spark installation 1.4 on hdp 2.3.4

ashneesharma88 — Mon, 13 Jun 2016 18:23:50 GMT

issue is with spark binary only.

link.:- https://issues.apache.org/jira/browse/PHOENIX-2287

Re: spark installation 1.4 on hdp 2.3.4

jyadav — Mon, 13 Jun 2016 18:40:00 GMT

@Ashnee Sharma I can see one patch Phoenix patch in bug https://issues.apache.org/jira/browse/PHOENIX-2287, If you are suing Phoenix then you probably need to apply it in your phoenix env.

Re: spark installation 1.4 on hdp 2.3.4

ashneesharma88 — Mon, 13 Jun 2016 20:21:31 GMT

can you brif me how to apply it in phoenix env?

Re: spark installation 1.4 on hdp 2.3.4

jyadav — Mon, 13 Jun 2016 20:22:03 GMT

Hi @Ashnee Sharma please accept this answer if you are good with the provided information to close this thread.

Re: spark installation 1.4 on hdp 2.3.4

jyadav — Mon, 13 Jun 2016 20:25:57 GMT

Sure, we will discuss this on this https://community.hortonworks.com/questions/39369/pheniox-patch-apply.html thread. Lets close this one.

Re: spark installation 1.4 on hdp 2.3.4

ashneesharma88 — Mon, 13 Jun 2016 21:43:46 GMT

I resolved the issue, with downgrade the version.

steps what I perform

stpe 1:-

cat /etc/yum.repos.d/HDP-TP.repo

#VERSION_NUMBER=2.3.2.0-2950

[HDP-2.3.2.0]

name=HDP Version - HDP-2.3.2.0

baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.3.2.0

gpgcheck=1

gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.3.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

[HDP-UTILS-1.1.0.20]

name=HDP Utils Version - HDP-UTILS-1.1.0.20

baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos7

gpgcheck=1

gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.3.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

step2:-

yum install spark_2_3_2_0_2950*

To check go to /usr/hdp/2.3.2.0-2950/spark/sbin

Step 3:- start ./start-all.sh