Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

BDR failures after CM upgrade

Highlighted

BDR failures after CM upgrade

New Contributor

Hi,

 

We have our cluster configured to run BDR.Recently we upgraded our CM to 5.8.1 version. After that we noticed some BDR jobs constantly failing. Upon checking ,seems like these BDR jobs ran during the CM upgrade time and  have directories created with hdfs ownership on the time of CM upgrade ,causing failures with permission denied for user .

Is this an expected behaviour? .Please advise.

 

Thanks,

Naigy

3 REPLIES 3
Highlighted

Re: BDR failures after CM upgrade

Cloudera Employee

Hi Naigy,

 

No, this is not really expected. HDFS BDR jobs are done by a modified version of distcp (called enterprise-distcp) that is included with CM. But as with regular distcp, it will submit a regular YARN/MapReduce job, and as such, a CM upgrade shouldn't disturb running hadoop services, including this.

 

Can you please clarify the CM version you are upgraded from?

Have you seen any other irregularities during or after the upgrade?

 

 

cheers,

zegab

Highlighted

Re: BDR failures after CM upgrade

Super Guru

The assumption here is that upgrading caused a failure.  We could review the information you used to arrive at that conclusion and help make sure that is the correct cause.  Can you share with us the errors that you saw?

 

Thanks!

Highlighted

Re: BDR failures after CM upgrade

New Contributor

Hi,

 

The error occured in the BDR failed logs were permission denied due to hdfs ownership on folders created in target cluster.

"org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=username, access=WRITE, inode=""/prod/view/warehouse/***/year=2016/month=10/day=12"":hdfs:accounts:drwxr-xr-x"

 

I have noticed the same issue in around 6 BDR jobs scheduled in the CM. All this ran during the time of upgrade window 13th october 18:00-23:00. Thats how came to conclusion that upgrade might be a cause.

 

Thanks,

Naigy

Don't have an account?
Coming from Hortonworks? Activate your account here