Despite extensive efforts, I was unable to directly resolve the issue, but I devised a workaround. Rather than directly accessing the Hadoop Job object for status updates, I extracted the job ID after submitting the job. Using this job ID, I created an ApplicationID object, which I then used to instantiate my YarnClient. This approach enabled me to effectively monitor the status and completion rate of the running job. Interestingly, both my distcp job and YarnClient are utilizing the same HadoopConf object (YarnClient is instantiated right after the DistCP Job is executed) and is within the same scope of the UserGroupInformation. The exact reason why the YarnClient can access the necessary information while the Job object cannot remains unclear. Nevertheless, this workaround has successfully unblocked me. Additional context: I am using Java 8 and running on an Ubuntu Xenial image. ... View more

Hi I'm also facing this same exact issue. I can submit the distcp job to yarn which runs and completes but I'm not able to check the progress of the job while it's in running. The strange thing is that there's no issue when I use the hadoop2 library. This issue came up as I upgrade to hadoop version > 3.2.3 Where you able to resolve your issue?   ... View more