Reply
RK
Explorer
Posts: 17
Registered: ‎09-24-2015

Re: IOException All datanodes DatanodeInfoWithStorage ,DISK] are bad. Aborting

Grep on all 3 nodes involved in this operation did not have any string matched for XceiverCount and for "exceeds the limit of concurrent xceivers". 

 

Looks like pastebin link is expired, can you add it again, I will post for the duration. Did not see anything unusual though.

 

Thanks

Cloudera Employee
Posts: 25
Registered: ‎11-22-2017

Re: IOException All datanodes DatanodeInfoWithStorage ,DISK] are bad. Aborting

You can go to the link again and click on "+ new paste" for a new text field to post the logs. Once done, scroll below and click on "create new paste". A link will be generated. Share that link with us.

Highlighted
Expert Contributor
Posts: 338
Registered: ‎01-25-2017

Re: IOException All datanodes DatanodeInfoWithStorage ,DISK] are bad. Aborting

Hey,

 

Once the job failed, the disk space disappear?

 

Can you check if the disk space occur on the application master nodes?

 

I assume this is the container logs and you can check this while the job running.

RK
Explorer
Posts: 17
Registered: ‎09-24-2015

Re: IOException All datanodes DatanodeInfoWithStorage ,DISK] are bad. Aborting

here are links to pastebin which has excerpts from log files.

There are one container log and other 3 datanode logs which were part of the same pipeline write operation.


container log
https://pastebin.com/SbMvr52W

node 1
https://pastebin.com/hXbNXCe1

node 2
https://pastebin.com/qAMfkVsg

node 3
https://pastebin.com/RL5W2qfp

 

Thanks

RK
Explorer
Posts: 17
Registered: ‎09-24-2015

Re: IOException All datanodes DatanodeInfoWithStorage ,DISK] are bad. Aborting

This is not due to disk space, as there is sufficient disk space even when this job is running.

 

Thanks

RK

Announcements