Writing to hdfs using rest api pausing spark application

[#1]

Hi guys,

I am running a spark application which writes some information to log files from inside every reduceByKey() operation. For this i am using webhdfs rest api and writing to log files in hdfs. The problem is when i write from each reduce operation it runs fine till certain number of finished tasks but pauses after that. But if i don't write to hdfs from each reduce operation and rather just perform reduction everything runs fine. The data size is probably just 50 bytes max. So is there a limit to how many connections a webhdfs server can simultaneously have open ? If not what can be the problem ? I am stuck out here so some direction would be helpful.

[#2]

What is your HDP version?

[#3]

HDP-2.4.2.0-258 is my current version that i am using