Member since
10-18-2016
9
Posts
0
Kudos Received
0
Solutions
11-11-2016
04:58 PM
Thank you!
... View more
11-01-2016
08:42 AM
Hello, I am planning to enable the HDFS Storage-Balancer as per this article: https://www.cloudera.com/documentation/enterprise/5-5-x/topics/admin_dn_storage_balancing.html. I plan to use the defaults, but change the "dfs.datanode. available-space- volume-choosing- policy.balanced- space-preference- fraction" to 1.0 (understanding that this could result in some bottlenecks. My questions about this process are: 1) We are also running HBase in this cluster, with RegionServers colocated with the Datanodes. Are there any gotchas that should be considered given this? 2) We don't have CM Enterprise, so the official rolling restart is not available to us. Is it possible to restart the Datanode role on nodes individually? Thank you.
... View more
Labels:
- Labels:
-
HDFS
10-20-2016
07:03 AM
Thank you. This is a great feature, and I appreciate the link. Unfortunately, our cluster is running 5.7.1, and given my lack of experience with CDH (I inherited this cluster) I'm loathe to upgrade it at the moment.
... View more
10-18-2016
07:03 AM
Hi. Some of my datanodes have different disk size. For example: /dev/sdc1 918G 384G 534G 42% /data/disk1 /dev/sdd1 459G 381G 78G 84% /data/disk2 /dev/sde1 459G 391G 69G 86% /data/disk3 /dev/sdf1 459G 389G 70G 85% /data/disk4 My understanding is that there is currently no functionality for balancing within a datanode, so I'd have to move data around manually. I've found this article on performing the procedure: http://www-01.ibm.com/support/docview.wss?uid=swg21702775 (Procedure 1). Has anyone actually done this (or something similar)? Can you share any issues/caveats you ran across? Is this the best way to do it? If the other 3 disks fill up, will that datanode continue to write to disk1? Thank you.
... View more
Labels:
- Labels:
-
HDFS