spooldir channel error - too many files. - how to replay the good files?

Question

so i have a flume agent with a spooldir channel that got too full (too many files error).

the fix was to move the files out of the spooldir, move the checkpoint, and restart the agent, resulting in data flowing again.

but now i have some flume-formatted data that i would like to ingest.

can i move the files into the spooldir 10% at a time fx?

would i have to read the files and add them to the flume source?

additional notes:

the reason for the spool dir was a missing sink (another flume agent- turned off)

pdvorak · Accepted Answer

Moving them in 10% at a time would be a good plan. You'll want to make sure that they are on the same filesystem, just in a different directory, so the move isn't copying across filesystems, just changing inodes.

Going forward, it would be recommended to add a flume channel trigger, to alert you when the channel starts filling up, if your downstream agent isn't accepting events.

-pd

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.