Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Possible to drain a Flume channel?

SOLVED Go to solution
Highlighted

Possible to drain a Flume channel?

New Contributor

If I were planning on decommisioning a node, is it possible to drain the flume channels? If I disable the sink, Flume doesn't proceed to do anything and if I change the batch size on the Kafka source to 0, it still fetches data.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Possible to drain a Flume channel?

Super Collaborator
What is your channel size in the flume metrics page reported as? Is it decreasing?

flume keeps at least the two most recent log files in the flume file channel at all times, regardless of whether it is fully drained or not. The best is to review the channel size in the flume metrics page, or on the channel size charts.

-pd
4 REPLIES 4

Re: Possible to drain a Flume channel?

Super Collaborator
The recommended path in this situation is to just comment out the sources line that specifies which sources are configured.:

# tier1.sources = kafkasource1 kafkasource2 etc

The flume agent can function without any sources and will then drain the channel through the sinks, without adding any new data to the channel.

-pd

Re: Possible to drain a Flume channel?

New Contributor

Maybe it is draining in my case and keeping old data behind until its rotated? I commented out the source and a few minutes later it created log-305 with no data.

 

-rw-r--r-- 1 flume flume    0 Jan 17 05:47 in_use.lock
-rw-r--r-- 1 flume flume 1.6G Jan 17 22:10 log-302
-rw-r--r-- 1 flume flume   47 Jan 17 22:10 log-302.meta
-rw-r--r-- 1 flume flume 1.6G Jan 17 22:14 log-303
-rw-r--r-- 1 flume flume   47 Jan 17 22:13 log-303.meta
-rw-r--r-- 1 flume flume  99M Jan 17 22:14 log-304
-rw-r--r-- 1 flume flume   47 Jan 17 22:17 log-304.meta
-rw-r--r-- 1 flume flume    0 Jan 17 22:17 log-305
-rw-r--r-- 1 flume flume   47 Jan 17 22:17 log-305.meta

Re: Possible to drain a Flume channel?

Super Collaborator
What is your channel size in the flume metrics page reported as? Is it decreasing?

flume keeps at least the two most recent log files in the flume file channel at all times, regardless of whether it is fully drained or not. The best is to review the channel size in the flume metrics page, or on the channel size charts.

-pd

Re: Possible to drain a Flume channel?

New Contributor

It does decrease but I was monitoring the size of the directory and expected to be close to empty when done draining. It's good to know it just keeps the last two logs and I also noticed it creates a new empty checkpoint when its done. Thanks for the help!