Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spooldir channel error - too many files. - how to replay the good files?

Solved Go to solution

spooldir channel error - too many files. - how to replay the good files?

Explorer

so i have a flume agent  with a spooldir channel that got too full (too many files error).

 

the fix was to move the files out of the spooldir, move the checkpoint, and restart the agent, resulting in data flowing again.

 

but now i have some flume-formatted data that i would like to ingest.

can i move the files into the spooldir 10% at a time fx?

would i have to read the files and add them to the flume source?

 

 

additional notes:

the reason for the spool dir was a missing sink (another flume agent- turned off)

1 ACCEPTED SOLUTION

Accepted Solutions

Re: spooldir channel error - too many files. - how to replay the good files?

Super Collaborator
Moving them in 10% at a time would be a good plan. You'll want to make sure that they are on the same filesystem, just in a different directory, so the move isn't copying across filesystems, just changing inodes.

Going forward, it would be recommended to add a flume channel trigger, to alert you when the channel starts filling up, if your downstream agent isn't accepting events.

-pd
2 REPLIES 2

Re: spooldir channel error - too many files. - how to replay the good files?

Super Collaborator
Moving them in 10% at a time would be a good plan. You'll want to make sure that they are on the same filesystem, just in a different directory, so the move isn't copying across filesystems, just changing inodes.

Going forward, it would be recommended to add a flume channel trigger, to alert you when the channel starts filling up, if your downstream agent isn't accepting events.

-pd

Re: spooldir channel error - too many files. - how to replay the good files?

Explorer

thanks for your answer,

do i need the .meta and .meta.tmp files?