Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎02-22-2014
Accepted Solution

Increase Flume graceful restart time

I'm using Flume's HDFS sink and it sometimes doesn't close the temporary files in HDFS, so that these data become lost. I think it is because supervisord doesn't give Flume agents enough time to stop, since the default stopwaitsecs is 30 seconds, and after that supervisord just "kill -9" it.

 

WARN killing '388-flume-AGENT' (7994) with SIGKILL

 

If I kill the process in terminal, it'll take a minute or so to fully stop, thus making no unrenamed temporary files. I'm wondering whether it's possible to increase the stopwaitsecs. Thanks.

Posts: 152
Topics: 8
Kudos: 15
Solutions: 16
Registered: ‎07-16-2015

Re: Increase Flume graceful restart time

I believe this wait time of 30s is hard coded into the cloudera agent.

I don't think we can alter it other than doing a real dirty modification which I wouldn't recommend.

 

regards,

Mathieu

 

Announcements