Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Highlighted

After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Expert Contributor
 
11 REPLIES 11

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Expert Contributor

Initially, we do have 7 nodes and later added 5 more nodes. Each node of size 36TB of 10 disks and 256GB RAM. Befrore adding new nodes, to 74GB of data it is taking below 10 mins, After adding new nodes, it took 2 hrs to copy data into hdfs.

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Guru

can you check if rebalancing as kicked off or not

you should be able to grep this on namenode "org.apache.hadoop.hdfs.server.balancer.Balancer"..if the balancer has kicked off and is in progress, the process will be visible

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Are you measuring the time taken by a single writer to transfer 74GB into HDFS? 12x degradation sounds like something is wrong.

Check the NameNode web UI to see if HDFS looks healthy and all DataNodes are heart-beating regularly. Also check your client,NameNode and DataNode logs for exceptions.

Are you running balancer? If so check the balancer settings to ensure they are not too aggressive.

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Expert Contributor

I am able to see slow block reciever rate on datanode logs.

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Can you share the exact message you see?

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Expert Contributor

@Arpit Agarwal when i see the slaves file, i didn't see the newly added nodes. We are planning to add new hosts and restart the HDFS ad NN service.

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Super Guru
@Ram D

Two things come to mind. First one is mentioned by both @srai and @Arpit Agarwal and that is if balancer has kicked in. This is very likely and might be taking up more bandwidth in the cluster.

Another question that comes to mind is "are these new nodes in the same rack?". May be before you had all nodes in one rack but now you have cross rack trafiic also. Hadoop is rack aware so when you are adding new data, blocks are copied over to nodes in another rack also to ensure availability in case of rack failure.

Now if new nodes are in a different rack and balancer is also running, then that would explain why you are seeing slow performance.

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Expert Contributor

Those are in the same rack.

Re: After addig new nodes into the cluster, copying from local to HDFS is taking more time. What may be the reason?

Guru

@Ram D , would you consider sharing your findings and how you were able to find the problem, following the recommendations ?

Don't have an account?
Coming from Hortonworks? Activate your account here