Reply
New Contributor
Posts: 1
Registered: ‎01-03-2018

Maximum capacity per DataNode

 Is there any upper limit for maximum capacity per node? Can data nodes scale to more than 100TB/node? 

Posts: 1,748
Kudos: 365
Solutions: 277
Registered: ‎07-31-2013

Re: Maximum capacity per DataNode

There are no limits in the source code implementation, if that is what you are asking. There are practical limits such as replication bandwidth (applied at loss) and reporting load (for low-latency operations) that you will run into when exceeding storage boundaries.

See also our Hardware Requirements guide: https://www.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide.h...
New Contributor
Posts: 1
Registered: ‎03-20-2018

Re: Maximum capacity per DataNode

Hi,

 

For a Data node with 100TB of size, how much RAM is required ??

Cloudera Employee
Posts: 42
Registered: ‎08-16-2016

Re: Maximum capacity per DataNode

That's mostly a function of blocks stored on a DataNode. For example, a rule of thumb is one GB heap size for DN for every one million blocks stored on that DN.

Posts: 1,748
Kudos: 365
Solutions: 277
Registered: ‎07-31-2013

Re: Maximum capacity per DataNode

Agreed. You shouldn't need more than 3-4 GiB of heap, going by an x3 or x4
factor of ideal block count for that storage (storage divided by block
size).
Announcements