We have our cluster configured to run BDR.Recently we upgraded our CM to 5.8.1 version. After that we noticed some BDR jobs constantly failing. Upon checking ,seems like these BDR jobs ran during the CM upgrade time and have directories created with hdfs ownership on the time of CM upgrade ,causing failures with permission denied for user .
Is this an expected behaviour? .Please advise.
No, this is not really expected. HDFS BDR jobs are done by a modified version of distcp (called enterprise-distcp) that is included with CM. But as with regular distcp, it will submit a regular YARN/MapReduce job, and as such, a CM upgrade shouldn't disturb running hadoop services, including this.
Can you please clarify the CM version you are upgraded from?
Have you seen any other irregularities during or after the upgrade?
The assumption here is that upgrading caused a failure. We could review the information you used to arrive at that conclusion and help make sure that is the correct cause. Can you share with us the errors that you saw?
The error occured in the BDR failed logs were permission denied due to hdfs ownership on folders created in target cluster.
"org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=username, access=WRITE, inode=""/prod/view/warehouse/***/year=2016/month=10/day=12"":hdfs:accounts:drwxr-xr-x"
I have noticed the same issue in around 6 BDR jobs scheduled in the CM. All this ran during the time of upgrade window 13th october 18:00-23:00. Thats how came to conclusion that upgrade might be a cause.