Code Repositories
Find and share code repositories
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)
Expert Contributor
Repo Description

Background:

To run Hadoop distcp command on a Cluster with NameNode High Availability (HA) enabled, the following is required:

* Adding of nameservice information of both Source and destination cluster

* Restarting of the services.

The reason being that YARN ResourceManager renews delegation tokens for applications.

Solution:

To avoid server side configuration, the MapReduce jobs can send the configurations to RM at runtime and RM uses these configurations to renew tokens via mapreduce.job.send-token-conf

We can leverage the same via Oozie Distcp Action. Git Repo contains Oozie distcp Action template that would allow basic oozie distcp action on a Kerberos environment and help parameterize on runtime. This way end users can run at their schedule.

  1. job.properties
  2. workflow.xml
Repo Info
Github Repo URL https://github.com/saumilmayani/oozie-distcp_template.git
Github account name saumilmayani
Repo name oozie-distcp_template.git
328 Views
0 Kudos
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎02-27-2018 07:04 PM
Updated by:
 
Contributors
Top Kudoed Authors