Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Expert Contributor

We have seen performance issue and stability issue in heavy workload system, this is due to disk latency/Shared Disk. I.e frequent namenode failover, longer boot time, slower checkpoint, slower logging, Higher fsync will cause session expiry, etc.

We never recommend a shared disk for Namenode, Journal node and Zookeeper. All of these services should have a dedicated disk.

You can configure following disk type according to your HDFS workload.
[dfs.namenode.name.dir]
 Namenode fsimage directory, dedicated disk => HDD 15K RPM 
[dfs.journalnode.edits.dir] 
 Namenode edit log directory, dedicated disk => SSD 
[dataDir from zoo.cfg]
 Zoookeeper snapshot and transaction logs, for normal usage from ZKFC and HBase => HDD 15K RPM 
 Zoookeeper snapshot and transaction logs, If it used by Nifi/ Accumulo/ Kafka/ Storm/ HBase and ZKFC => SSD 

If you are using RAID for meta-directory(dfs.namenode.name.dir & dfs.journalnode.edits.dir), then disable RAID and check the Non-RAID performance. There is a strong redundant for meta-directory (fsimage & edits are available from Standby NameNode and remaining QJNs).

If RAID is not disabled for JN, then consider using different RAID. RAID 1 and RAID 10 are also good for the dfs.journalnode.edits.dir set rather than RAID 5, due to increased in write latency for small block writes.

if you don't have a faster disk, then don't consider using fsimage replication. It will impact write performance even if one of the disks slower.

2,290 Views