Reply
Contributor
Posts: 106
Registered: ‎08-07-2017

Migration from mapreduce MRV1 framework to yarn in CDH 5.9

[ Edited ]

Hello experts,

 

We are currently using mapreduce Mrv1 framework for job processing and wish to migrate to YARN in CDH5.9.2 cluster.

 

I came across https://www.cloudera.com/documentation/enterprise/5-3-x/topics/cdh_ig_mapreduce_to_yarn_migrate.html documentation link when I searched. But Its not clear to exactly what steps need to be followed as we will be doing migration on production cluster itself.

 

The above mentioned document describes the parameters different in mrv1 and yarn .

 

If  you can please help me with the steps need to be followed, that would be great.

 

Thanks,

Priya

Posts: 1,730
Kudos: 356
Solutions: 273
Registered: ‎07-31-2013

Re: Migration from mapreduce MRV1 framework to yarn in CDH 5.9

If you use Cloudera Manager, you will find it pop up a wizard for you when you try to add YARN into a cluster that already carries MapReduce (v1). This wizard will take care of some cluster-side configuration migration (such as setting up resource values based on current slot counts, host data, etc.).

The link you've posted relates to users of CDH who use the CLI to manage their cluster. For CM users that document does not apply.

If your question is developer-oriented, most of your job jars as long as they were compiled against CDH5 MRv1 will continue to function as is - just ensure you never bundle the MR dependencies into the app jar (i.e. fat jar) as this could cause classpath issues when upgrading or when moving from MRv1 to YARN+MRv2.

As an admin that uses CM, just follow https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_yarn_service.html#concept_qgv_...
Contributor
Posts: 106
Registered: ‎08-07-2017

Re: Migration from mapreduce MRV1 framework to yarn in CDH 5.9

@Harsh J,

 

Thank you for inputs. I am usig CM for managing the cluster. The steps mentioned in the link you mentioned https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_yarn_service.html#concept_qgv_... will be same for 5.9 version as well or not?

 

As part of migration from MRv1 to YARN , I added service to the cluster through add service option.

 

But one of our important application job was not running after addition of YARN service to the cluster. 

 

Thanks,

Priya

Posts: 1,730
Kudos: 356
Solutions: 273
Registered: ‎07-31-2013

Re: Migration from mapreduce MRV1 framework to yarn in CDH 5.9

Yes that document applies to 5.9 too.

The migration tries to take care of common settings but its still a good
idea to double-check the values it specifies. See also the document and
spreadsheet available at
https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html
if
you want a more thorough cluster tuning procedure for your new YARN service.

When you say 'not running' do you mean a hang in ACCEPTED state or such, or
an actual error? Best to open a separate topic for that unless its already
resolved.
Highlighted
Contributor
Posts: 106
Registered: ‎08-07-2017

Re: Migration from mapreduce MRV1 framework to yarn in CDH 5.9

Hi Harsh,

 

Thanks for inputs. Will cross-check the values.

 

 

Thanks,

Priya

Contributor
Posts: 106
Registered: ‎08-07-2017

Re: Migration from mapreduce MRV1 framework to yarn in CDH 5.9

@Harsh J,

 

I refered to cloudera this https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.htmldocument, all the parameters are specified except yarn.scheduler.minimum-allocation-mb.

 

And the value for Java Heap Size of NodeManager in Bytes this parameter is too low i.e. 50MB. Other values are good.

 

Can you please tell me whether this is what causing issue?

 

Thanks,

Priya

 

Contributor
Posts: 106
Registered: ‎08-07-2017

Re: Migration from mapreduce MRV1 framework to yarn in CDH 5.9

Hi Harsh,

 

I have one more query. We are not doing CDH upgrade, just migrating from mapreduce to yarn, so do we have to Import MapReduce Configuration?

Please help.

Thanks,
Priya

Announcements