HDP3: hadoop distcp NullPointerException

[#1]

Below command failed on HDP3. Any idea? Thanks a lot.

sudo -u hadoop hadoop distcp hdfs://namenode.mycluster.com:8020/tmp/txt hdfs://namenode.mycluster.com:8020/tmp/txt2


java.lang.NullPointerException
    at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.stop(TimelineV2ClientImpl.java:563)
    at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.serviceStop(TimelineV2ClientImpl.java:141)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
    at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:495)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
    at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
    at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
    at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158)
    at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1868)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1309)
    at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
    at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:202)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
2018-11-02 18:57:15,753 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
    at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:979)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
    at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.NullPointerException
    at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:177)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
    ... 14 more
2018-11-02 18:57:15,754 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException

[#2]

Add below properties to yarn-site.xml:

<property><name>yarn.resourcemanager.webapp.address.rm1</name>
<value>hostname_of_resourcemanager_1:8088</value>
</property>
<property><name>yarn.resourcemanager.webapp.address.rm2</name>
<value>hostname_of_resourcemanager_2:8088</value>
</property>
<property><name>yarn.resourcemanager.webapp.https.address.rm1</name>
<value>hostname_of_resourcemanager_1:8090</value>
</property>
<property><name>yarn.resourcemanager.webapp.https.address.rm2</name>
<value>hostname_of_resourcemanager_2:8090</value>
</property>

After adding these properties, restart resourcemanager and nodemanager.

In the above properties, rm ids (rm1 and rm2) are values given for yarn.resourcemanager.ha.rm-ids property in yarn-site.xml. If you have given value other than rm1,rm2 for yarn.resourcemanager.ha.rm-ids property then make changes in the above mentioned webapp properties accordingly.