Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

DistCp failure during commit

DistCp failure during commit

Super Collaborator
Hi,
 
DistCp failed during commit, i suspect from what i see on privillages, any insights?
 
2017-02-05 09:34:42,504 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:cloudera-scm (auth:SIMPLE)
cause:java.io.IOException: Was asked to shut down.
2017-02-05 09:34:42,504 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Was asked to shut down.
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1546)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1707)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1540)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1473)
2017-02-05 09:34:42,506 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
5 REPLIES 5

Re: DistCp failure during commit

Champion
It failed to even launch the AM for the job against YARN. There is a reconfiguration somewhere. Are you able to launch an MR job outside of distcip? From another hosts? Do other services work like HDFS from this host?

Re: DistCp failure during commit

Super Collaborator

All others are working fine

Highlighted

Re: DistCp failure during commit

Champion
Can you share the yarn-site.xml and mapred-site.xml from the host that is running this distcp job?

Can you post the distcp command?

Re: DistCp failure during commit

Super Collaborator

Even it failed on diiferent node? do you need specific definition from the xmls?

Re: DistCp failure during commit

Super Collaborator

Any other ideas?