Reply
Highlighted
Explorer
Posts: 7
Registered: ‎12-29-2018
Accepted Solution

HDFS sizing and the right model

Good day guys, im newby in Cloudera and wanted to ask 2 questions.

 

1) I got 20TB of data and i should migrate it to 10 servers, do i need to have 20TB of disk on each server ?

2) How do i organize the right HDFS model (NameNode, DataNode, SecondaryNameNone) on those 10 servers ?

 

Thanks, i hope to receive the answer very soon )

 

 

Expert Contributor
Posts: 108
Registered: ‎02-23-2018

Re: HDFS sizing and the right model

Hi @Adilm,

 

 1) If you want to migrate all data, you can compress them and allocated in other nodes/servers. And not need 20TB of disk.

     Althow if you need availble the data information, yo have 2 scenarios: 

         - Ten replication factor: then need 20TB per server.

         - One replication factor: only need 20TB distributed in 10 servers.

         - Best: replication factor 5 and 4TB per server.

 

2) Its depends, you need one namenode, one secondarynamenode, and for example 8 datanodes. You need to put attention of resources of your hosts.

 

 

 

Regards,

Manu.

 

 

 

Explorer
Posts: 7
Registered: ‎12-29-2018

Re: HDFS sizing and the right model

Thanks for your reply, so if i get it the right way, size on each server depends on replication factor i put, is there any table of dependencies of replication factor and disk sizing ?

 

Also wanted to ask about the resources on each node, so summary i need some documentation about replica factor, sizing and ram usage.

 

 

Expert Contributor
Posts: 108
Registered: ‎02-23-2018

Re: HDFS sizing and the right model

Hi @Adilm,

 

You are right. There are not any table, you must to study your scenario(HA, security, access number ...).

 

Some questions:

    - Volume users?

    - Volume data?

 

All documentation is available here, according your version:

  https://www.cloudera.com/documentation/enterprise/latest.html

    

 

 

Regards,

Manu.

Explorer
Posts: 7
Registered: ‎12-29-2018

Re: HDFS sizing and the right model

Thanks )