I am trying to install a Hadoop cluster on a Dell blade server with 12 core dual CPU, 32 gb ram with VMware exsi on it. How many node can install on it?

[#1]

I am trying to install a hadoop multinode cluster. So i had a 12(24 vcores) dual core processor and 32 gb ram. I am thinking to create a cluster with 6 nodes by allocating 4 vcores and 8-16 gb ram for each node. My questions is can i do that. Would Vm able to manage if i use all the memory. Can any body suggest me the best possible way of allocating the above resources.

[#2]

Hi! You can overcommit ram in esxi so this will work. Each VM will use what it needs up to the max. However since HDP tends to run at full performance you will run out of ram at some point. From a hardware point of view your server needs more ram. Most esxi servers with dual 12 core CPU will need 128G of ram to fully utilize your CPU resource.

[#3]

@R c does this answer your question? thanks, Ron

[#4]

can you tell me what would be the best configuration like how many nodes can fit into the current hardware and how can i place masters and nodes

[#5]

@Ron Lee

You can install 3 node cluster easily.

1) Would you let me know what component you required.

2) For for what purpose you are going to install the cluster? i.e for testing or for development.

[#6]

I Want Hive,oozie,hbase,saprk, kafka, storm (I am not using kafka and storm currently but would be great if it fits). I currently creating for like testing.

[#7]

@Ron Lee

If data is not much then you can run same configuration with 3 Node cluster on blade server.

[#8 · Accepted solution]

@R c

You state that your server has 12 cores (2 threads per core) and 32 GB RAM. You did not state whether your server is x86. If it is a Power7 CPU then 12 cores is a lot of CPU power. I'll assume is x86.

Then you state that you would like 6 nodes with 4 vCPU per node and 8-16 GB per each. The RAM math does not add-up. You would need 48-96 GB to meet that wish.

You did not mention anything about the type of storage that you could use. Is it the internal storage of the server, NAS or SAN? How much can you use? For this type of cluster, you would need 25-30 GB for root and logs and separate storage for data nodes, whatever you can afford, but not less than 50-100 GB/data node if you really want to have a big data capable for development cluster.

If your resources are really 12 cores (24 vCPU) and 32 GB RAM, that is a really not a great RAM/vCPU ratio. A good rule of thumb is to have minimum 4 GB RAM for each vCPU. That means in your case, it would have been good if you had 96 GB RAM. My response would have been easily: 6 nodes as you wanted, each 4vCPU and 16 GB RAM. I would never allocate less than 8 GB of RAM per node.

If you RAM is really 32 GB then, maybe a 4-node cluster which will be memory bound. HBase, Spark and Hive LLAP may have some limitations.

Keep in mind that your resources are a reasonable fit for a very small POC/dev environment, however, useable.

+++

If any of the responses was helpful, please vote and accept best answer.

[#9]

can i allocate 10gb ram for each node

[#10]

@R c

Of courseOf course you can, it is not ideal, but it is useable.

If it helped, pls vote/accept best answer.