we are running cdh 5.10 cluster with about 500 nodes, we installed nn/jn/zk as bellow:
host1: namenode, journal node, zookeeper
host2: namenode, journal node, zookeeper
host3: journal node, zookeeper
(no other service is install on these three host)
namenode's fsimage & editlog storage dir is configed at "/data1/dfs/nn/" and "/data2/dfs/nn/"
journal node's editlog storage dir is configed at "/data3/dfs/nn/"
/data1 and /data2 is mount at individual disk driver
when we look into the namenode's log, we found that namenode takes long time to flush editlog to journalnode, and flush to local disk driver is not taking that much time.as seeing from the log, flush to journal node is about 4x~5x times longer than flush to local disk. bellow is some log snippets:
--snippet 1
2017-08-15 09:50:12,946 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 326245 Total time for transactions(ms): 3997 Number of transactions batched in Syncs: 284851 Number of syncs: 41371 SyncTimes(ms): 14194 3543 3397
2017-08-15 09:51:12,951 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 533598 Total time for transactions(ms): 6599 Number of transactions batched in Syncs: 471796 Number of syncs: 61757 SyncTimes(ms): 21695 5450 5132
2017-08-15 09:52:12,951 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 692744 Total time for transactions(ms): 11074 Number of transactions batched in Syncs: 610178 Number of syncs: 82561 SyncTimes(ms): 31668 7356 6787
2017-08-15 09:53:12,953 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 843165 Total time for transactions(ms): 14348 Number of transactions batched in Syncs: 742316 Number of syncs: 100838 SyncTimes(ms): 40082 10075 8269
2017-08-15 09:54:15,374 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 920690 Total time for transactions(ms): 15248 Number of transactions batched in Syncs: 808311 Number of syncs: 110467 SyncTimes(ms): 43884 39289 10217
2017-08-15 09:54:30,821 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 920690 Total time for transactions(ms): 15248 Number of transactions batched in Syncs: 810222 Number of syncs: 110468 SyncTimes(ms): 43910 39428 38668
2017-08-15 09:55:30,821 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 206372 Total time for transactions(ms): 2562 Number of transactions batched in Syncs: 160222 Number of syncs: 46144 SyncTimes(ms): 16389 5156 3410
--snippet 2
2017-08-15 09:59:14,716 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 175114 Total time for transactions(ms): 2762 Number of transactions batched in Syncs: 89590 Number of syncs: 85521 SyncTimes(ms): 29138 6897 5872
2017-08-15 10:00:14,716 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 221609 Total time for transactions(ms): 3499 Number of transactions batched in Syncs: 112617 Number of syncs: 108989 SyncTimes(ms): 38074 9056 7451
2017-08-15 10:01:45,172 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 251986 Total time for transactions(ms): 4000 Number of transactions batched in Syncs: 130562 Number of syncs: 121108 SyncTimes(ms): 42776 44831 14277
is this editlog sync behave normal? how can we speed up the flush operation when flush to journal node？
Any reply is appreciated.
