how to debug disk writes slowness and make it faster

[#1]

I am facing problem where disk writes are very slow due to heavy write operations.

Using rsync I am copying logs to logserver and logserver is not able to do faster writes.

Any suggestions to make log writes faster?

Req. is to have 600+ rsync process to copy logs to logserver and each log folder of more then 8gb and more then 9k files in each log folder.

[#2 · Accepted solution]

@vpemawat

If you don't change anything from your process and logging approach (e.g. separate workload to not compete for the same disks IOPS, timing etc), the only option left is SSD which will increase IOPS significantly. Even then is good to separate the workload to avoid contention. One of your challenges is driven by the quite high number of files written. If you would have used a tool like NiFi (or at least Flume) to ingest the logs and write lesser number output files and spread those to log folders across dedicated drives, then you could see some improvements. There is no magic bullet.