RouteOnAttribute Trigger when no file flows

Question

Hey, so I want to create a flow that creates a putemail trigger when NO files are present in a hdfs path for a certain date range:

my current flow is

GETHDFS->

ROUTEONATTRIBUTE(has the proper date matching criteria)

-->MATCHED(should be getting files here)

-->UNMATCHED(when this outputs zero I want to receive an email alert)

RouteOnAttribute Logic:

${Path:substring(0,6):equals(${now():toNumber():minus(86400000):format('yyyyMM')})}

Pretty much if I find no files within the current month(last 20 days) then I need to receive an email alert.

How do I get an alert when there are no files that fit the criteria?

@Former Member

Thank you matt sorry for yet another question.

MattWho · Accepted Answer

@fowler7878

You are correct that MonitorActivity can not be used to monitor on activity based on per month thresholds.

Your use case is not one NiFi is designed to handle. NiFi is designed to work with FlowFiles and typically NiFi processor are not designed to return directory listings.

Your use case may require you to build your own custom NiFi processor or perhaps your own custom script that your can execute via ExecuteStreamCommand or ExecuteScript processors where the script returns number of files in a target directory which you can then make notification routing decisions with.

I was just trying to offers a builtin solution/suggestion while not exactly what you are looking for.

Thank you,

Matt

MattWho · Answer

@fowler7878

I am not sure what you want to accomplish is doable via your design.

The GetHDFS processor will either find a file to get or it will produce no output FlowFiles. So when it finally does produce a FlowFile because it found a file, the path attribute I am sure will be something recent depending on how often you schedule the GetHDFS processor to run.

If you are only trying to notify someone if there has been a lapse of 20 days without getting a single file from HDFS, you may want to instead look at the monitorActivity processor. You would add this after your GetHDFS processor so it monitors for FlowFiles coming from your GetHDFS. You can configure it with a 20 day threshold. It can also notify when activity resumes after a lapse was reported.

Hope this helps you,

Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.