Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why were my drives not used

Why were my drives not used

New Contributor

A while back I built a Cloudera 4.x Hadoop Cluster using  Cloudera Manager.  The cluster runs fine and there are no issues; however when I was troubleshooting a bad drive indicated by smartctl I was curious to see if the drive was actually being used by Hadoop and to my suprise I found every node only using 1 drive out of 4.  Each node has 4 2T drives configured asJBOD and only the system drive was configured to be used as hdfs space?  I dont recall anything during the Automated install process that specified adding drives to HDFS?  Did I miss something or is this supposed to be part of the install?  I just assumed that Cloudera Manager used all the drives on each server during set up.  Luckily space has not been an issue yet but it will be soon.

2 REPLIES 2

Re: Why were my drives not used

The default directory used by the datanodes is /dfs/dn. If you want to use your four drives then the dfs.data.dir property should be modified to include the mount point of the four drives. Cloudera Manager will help you edit this, within the HDFS configuration
Regards,
Gautam Gopalakrishnan

Re: Why were my drives not used

Here's the documentation on how many automatic rules work, including which drives are automatically selected:
http://www.cloudera.com/content/cloudera/en/documentation/core/v5-3-x/topics/cm_mc_autoconfig.html

Perhaps they started with /dev or something, so we excluded them?

If you added the drives after initial configuration, then CM would not have picked them up either.
Don't have an account?
Coming from Hortonworks? Activate your account here