Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spooldir channel error - too many files. - how to replay the good files?

avatar
Contributor

so i have a flume agent  with a spooldir channel that got too full (too many files error).

 

the fix was to move the files out of the spooldir, move the checkpoint, and restart the agent, resulting in data flowing again.

 

but now i have some flume-formatted data that i would like to ingest.

can i move the files into the spooldir 10% at a time fx?

would i have to read the files and add them to the flume source?

 

 

additional notes:

the reason for the spool dir was a missing sink (another flume agent- turned off)

1 ACCEPTED SOLUTION

avatar
Moving them in 10% at a time would be a good plan. You'll want to make sure that they are on the same filesystem, just in a different directory, so the move isn't copying across filesystems, just changing inodes.

Going forward, it would be recommended to add a flume channel trigger, to alert you when the channel starts filling up, if your downstream agent isn't accepting events.

-pd

View solution in original post

2 REPLIES 2

avatar
Moving them in 10% at a time would be a good plan. You'll want to make sure that they are on the same filesystem, just in a different directory, so the move isn't copying across filesystems, just changing inodes.

Going forward, it would be recommended to add a flume channel trigger, to alert you when the channel starts filling up, if your downstream agent isn't accepting events.

-pd

avatar
Contributor

thanks for your answer,

do i need the .meta and .meta.tmp files?