Upgrade cloudera hadoop to 2.7

[#1]

Hi , My application runs on Hadoop 2.7 and Spark 2.1.1. So i upgraded the Spark to higher version(2.3) but I cant find anyway to upgrade the current installed hadoop version i.e 2.6 to 2.7. Can anyone please help me to understand if is that possible in cloudera to upgrade from lower hadoop version to required version. In the community i see the below comment. Just wanted to know if its still the case.

Question in community: Is not possible to upgrade to hadoop 2.7 form 2.6?

Ans in community: no there isn't as it has not be made part of the CDH distribution.

[#2 · Accepted solution]

@t5,

With CDH versions you get a base version of hadoop and then fixes and features on top of that.  Upgrading just one or two components is not supported and is likely to cause problems.

My suggestion would be to try your app against the latest version of CDH and see if it works.  If it doesn't, identify if the "break" is related to something that is specific to 2.7 that you need.

The answer is "no," you cannot upgrade to hadoop 2.7 in CDH; however, it is likely that you already have what you need in CDH 5.14. or 5.15.

[#3]

@t5,

We need some background about how you are using CDH in order to best answer your question.

- Why are you trying to upgrade the hadoop package to 2.7?

- Are you using CDH?  If so, which version?

- Are you using Cloudera Manager? If so, what version?

As for your request to upgrade, I assume there is some feature or bug fix that you want to attain.  Cloudera uses a base of hadoop 2.6 currently, but we maintain many fixes and features that have come since 2.6 in our CDH release.   In that sense, there is no real "hadoop 2.6" version in CDH; rather, it is hadoop 2.6 + a bunch of other stuff.

Due to the nature of these fixes, you need to keep your CDH versions consistent across all components.

If you have an issue and want to know in what CDH release it is fixed, please let us know.

Ben

[#4]

@bgooley Hi Thanks for your reply.

Let me put it this way my applications runs of Hadoop 2.7 and Spark 2.1.1. So what wil be best suitable CDH you can suggest?

Coming to my earlier question i am using the cloudera manager 5.15 and as you mentioned it is having hadoop 2.6 + fixes. Hence i was looking to upgrade to Hadoop 2.7

[#5 · Accepted solution]

@t5,

With CDH versions you get a base version of hadoop and then fixes and features on top of that.  Upgrading just one or two components is not supported and is likely to cause problems.

My suggestion would be to try your app against the latest version of CDH and see if it works.  If it doesn't, identify if the "break" is related to something that is specific to 2.7 that you need.

The answer is "no," you cannot upgrade to hadoop 2.7 in CDH; however, it is likely that you already have what you need in CDH 5.14. or 5.15.

[#6]

@bgooley Thanks. I am trying to deploy my application using the cloudera manager 5.15. Will see if my application works or not.

As i was going through the cloudera site and i came to know that CDH6 Beta version was available and it is having Hadoop 3. Was it advisable to use the CDH6 to just to crosscheck if my application works on that or not?

[#7 · Accepted solution]

@t5,

I am pretty sure we are still in the closed beta stage so you would have to enter into the beta program.  You can try contacting the folks responsible for that via our website:

https://www.cloudera.com/products/cloudera-enterprise-6.html

I would say there is a fair chance that your application works OK with CDH 5.15, though, so I'll stay optimistic!

[#8]

@bgooley Hi, Thanks a ton for all your response to my queries. I just deployed my application jar( post upgrading the Spark 1.6 to 2.2 using the CSD) and can see its working fine though i tested few basic scenarios. Let me create as a service and hope it will work fine as well.