Possible to drain a Flume channel?

[#1]

If I were planning on decommisioning a node, is it possible to drain the flume channels? If I disable the sink, Flume doesn't proceed to do anything and if I change the batch size on the Kafka source to 0, it still fetches data.

[#2]

The recommended path in this situation is to just comment out the sources line that specifies which sources are configured.:

# tier1.sources = kafkasource1 kafkasource2 etc

The flume agent can function without any sources and will then drain the channel through the sinks, without adding any new data to the channel.

-pd

[#3]

Maybe it is draining in my case and keeping old data behind until its rotated? I commented out the source and a few minutes later it created log-305 with no data.

-rw-r--r-- 1 flume flume    0 Jan 17 05:47 in_use.lock
-rw-r--r-- 1 flume flume 1.6G Jan 17 22:10 log-302
-rw-r--r-- 1 flume flume   47 Jan 17 22:10 log-302.meta
-rw-r--r-- 1 flume flume 1.6G Jan 17 22:14 log-303
-rw-r--r-- 1 flume flume   47 Jan 17 22:13 log-303.meta
-rw-r--r-- 1 flume flume  99M Jan 17 22:14 log-304
-rw-r--r-- 1 flume flume   47 Jan 17 22:17 log-304.meta
-rw-r--r-- 1 flume flume    0 Jan 17 22:17 log-305
-rw-r--r-- 1 flume flume   47 Jan 17 22:17 log-305.meta

[#4 · Accepted solution]

What is your channel size in the flume metrics page reported as? Is it decreasing?

flume keeps at least the two most recent log files in the flume file channel at all times, regardless of whether it is fully drained or not. The best is to review the channel size in the flume metrics page, or on the channel size charts.

-pd

[#5]

It does decrease but I was monitoring the size of the directory and expected to be close to empty when done draining. It's good to know it just keeps the last two logs and I also noticed it creates a new empty checkpoint when its done. Thanks for the help!