question Possible to drain a Flume channel? in Support Questions

Possible to drain a Flume channel?

eric2025 — Fri, 16 Sep 2022 14:04:36 GMT

If I were planning on decommisioning a node, is it possible to drain the flume channels? If I disable the sink, Flume doesn't proceed to do anything and if I change the batch size on the Kafka source to 0, it still fetches data.

Re: Possible to drain a Flume channel?

pdvorak — Thu, 17 Jan 2019 21:19:39 GMT

The recommended path in this situation is to just comment out the sources line that specifies which sources are configured.:

# tier1.sources = kafkasource1 kafkasource2 etc

The flume agent can function without any sources and will then drain the channel through the sinks, without adding any new data to the channel.

-pd

Re: Possible to drain a Flume channel?

eric2025 — Thu, 17 Jan 2019 22:23:34 GMT

Maybe it is draining in my case and keeping old data behind until its rotated? I commented out the source and a few minutes later it created log-305 with no data.

-rw-r--r-- 1 flume flume    0 Jan 17 05:47 in_use.lock
-rw-r--r-- 1 flume flume 1.6G Jan 17 22:10 log-302
-rw-r--r-- 1 flume flume   47 Jan 17 22:10 log-302.meta
-rw-r--r-- 1 flume flume 1.6G Jan 17 22:14 log-303
-rw-r--r-- 1 flume flume   47 Jan 17 22:13 log-303.meta
-rw-r--r-- 1 flume flume  99M Jan 17 22:14 log-304
-rw-r--r-- 1 flume flume   47 Jan 17 22:17 log-304.meta
-rw-r--r-- 1 flume flume    0 Jan 17 22:17 log-305
-rw-r--r-- 1 flume flume   47 Jan 17 22:17 log-305.meta

Re: Possible to drain a Flume channel?

pdvorak — Wed, 23 Jan 2019 18:49:59 GMT

What is your channel size in the flume metrics page reported as? Is it decreasing?

flume keeps at least the two most recent log files in the flume file channel at all times, regardless of whether it is fully drained or not. The best is to review the channel size in the flume metrics page, or on the channel size charts.

-pd

Re: Possible to drain a Flume channel?

eric2025 — Thu, 24 Jan 2019 02:18:04 GMT

It does decrease but I was monitoring the size of the directory and expected to be close to empty when done draining. It's good to know it just keeps the last two logs and I also noticed it creates a new empty checkpoint when its done. Thanks for the help!