Member since
06-08-2017
1
Post
0
Kudos Received
0
Solutions
06-08-2017
02:18 PM
Development environment IntelliJ 2017.1.4 / jdk 1.8.40 / scala 2.10.5 / spark 1.6.2 / hive 1.2.1 VM : virtualBox / 5 VM (ambari, master slave1, slave2, slave3) The reason is that the jar version is wrong.
I use HDP 2.4.3.0-227. I modified and solved the following. ================ modify bash_profile =============== add export SPARK_HOME=/usr/hdp/current/spark-client =================modify pom.xml======================== <repositories>
<repository>
<id>HDP</id>
<name>HDP Releases</name>
<url>http://repo.hortonworks.com/content/groups/public</url>
<!--url>http://repo.hortonworks.com/content/repositories/releases/</url-->
<layout>default</layout>
<releases>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
<checksumPolicy>warn</checksumPolicy>
</releases>
<snapshots>
<enabled>false</enabled>
<updatePolicy>never</updatePolicy>
<checksumPolicy>fail</checksumPolicy>
</snapshots>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.2.2.4.3.0-227</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.2.2.4.3.0-227</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.6.2.2.4.3.0-227</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>1.6.2.2.4.3.0-227</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_2.10</artifactId>
<version>1.6.2.2.4.3.0-227</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.4</version>
</dependency> <!-- Test -->
</dependencies> ====================================================
... View more