spark access old version of Hadoop 2.1.0 and Hive version 0.11

[#1]

Hi, All:

I’m trying to read and write from the hdfs cluster using SparkSQL hive context. My current build of spark is 1.5.2. The problem is that currently our company has very old version of hdfs (hadoop 2.1.0) and hive metastore (0.11) using Hortonworks bundle.

One of the possible solution is to establish a separate cluster running hadoop 2.6.0 and hive >0.12. But is it still possible I can read data from old hdfs cluster and write to new cluster ? Or the only solution is to upgrade my Hortonworks bundle ? Will that impact current Hive tables ?

If the only solution is to upgrade, will that be a risky process? If I just upgrade the hdfs and hive will that affect other components in the current cluster?

Here is the versions on the current hdfs cluster:

Hue

2.2.0:67

HDP

2.0.5

Hadoop

2.1.0

HCatalog

0.11.0

Pig

0.11.2

Hive

0.11.0

Oozie

4.0.0

Thanks a lot,

Jade

[#2 · Accepted solution]

@Jade Liu

that is old indeed, you have several options, either have a separate cluster and distcp data to new cluster, install earlier spark version on your current cluster, not supported by Hortonworks btw, upgrade the current cluster. With Ambari 2.2, we're dropping support for HDP 2.0 so we highly encourage you upgrade to the latest HDP, you will benefit immensely from all new features and fixes. Going from 2.0 to 2.3 is not supported, you have to upgrade to 2.1 first and then you can use Express Upgrade to go to 2.3. We recommend you contact Hortonworks support for upgrades of this kind.

[#3]

@Jade Liu how did you solve your problem? can you accept the best answer to close this thread or post your own solution?