Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Possible to drain a Flume channel?

avatar
Explorer

If I were planning on decommisioning a node, is it possible to drain the flume channels? If I disable the sink, Flume doesn't proceed to do anything and if I change the batch size on the Kafka source to 0, it still fetches data.

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
The recommended path in this situation is to just comment out the sources line that specifies which sources are configured.:

# tier1.sources = kafkasource1 kafkasource2 etc

The flume agent can function without any sources and will then drain the channel through the sinks, without adding any new data to the channel.

-pd

avatar
Explorer

Maybe it is draining in my case and keeping old data behind until its rotated? I commented out the source and a few minutes later it created log-305 with no data.

 

-rw-r--r-- 1 flume flume    0 Jan 17 05:47 in_use.lock
-rw-r--r-- 1 flume flume 1.6G Jan 17 22:10 log-302
-rw-r--r-- 1 flume flume   47 Jan 17 22:10 log-302.meta
-rw-r--r-- 1 flume flume 1.6G Jan 17 22:14 log-303
-rw-r--r-- 1 flume flume   47 Jan 17 22:13 log-303.meta
-rw-r--r-- 1 flume flume  99M Jan 17 22:14 log-304
-rw-r--r-- 1 flume flume   47 Jan 17 22:17 log-304.meta
-rw-r--r-- 1 flume flume    0 Jan 17 22:17 log-305
-rw-r--r-- 1 flume flume   47 Jan 17 22:17 log-305.meta

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

It does decrease but I was monitoring the size of the directory and expected to be close to empty when done draining. It's good to know it just keeps the last two logs and I also noticed it creates a new empty checkpoint when its done. Thanks for the help!