question Re: spooldir channel error - too many files. - how to replay the good files? in Support Questions

spooldir channel error - too many files. - how to replay the good files?

sbn — Fri, 16 Sep 2022 14:27:38 GMT

so i have a flume agent with a spooldir channel that got too full (too many files error).

the fix was to move the files out of the spooldir, move the checkpoint, and restart the agent, resulting in data flowing again.

but now i have some flume-formatted data that i would like to ingest.

can i move the files into the spooldir 10% at a time fx?

would i have to read the files and add them to the flume source?

additional notes:

the reason for the spool dir was a missing sink (another flume agent- turned off)

Re: spooldir channel error - too many files. - how to replay the good files?

pdvorak — Wed, 19 Jun 2019 14:50:44 GMT

Moving them in 10% at a time would be a good plan. You'll want to make sure that they are on the same filesystem, just in a different directory, so the move isn't copying across filesystems, just changing inodes.

Going forward, it would be recommended to add a flume channel trigger, to alert you when the channel starts filling up, if your downstream agent isn't accepting events.

-pd

Re: spooldir channel error - too many files. - how to replay the good files?

sbn — Thu, 20 Jun 2019 08:37:46 GMT

thanks for your answer,

do i need the .meta and .meta.tmp files?