09-08-2017 12:43 AM
I'm using Flume's HDFS sink and it sometimes doesn't close the temporary files in HDFS, so that these data become lost. I think it is because supervisord doesn't give Flume agents enough time to stop, since the default stopwaitsecs is 30 seconds, and after that supervisord just "kill -9" it.
WARN killing '388-flume-AGENT' (7994) with SIGKILL
If I kill the process in terminal, it'll take a minute or so to fully stop, thus making no unrenamed temporary files. I'm wondering whether it's possible to increase the stopwaitsecs. Thanks.
09-08-2017 02:50 AM
I believe this wait time of 30s is hard coded into the cloudera agent.
I don't think we can alter it other than doing a real dirty modification which I wouldn't recommend.