Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi expression - evaluate cron schedule and current timestamp?

Nifi expression - evaluate cron schedule and current timestamp?

New Contributor
I know and have used the standard Nifi generateFlowFileProcessor running on a cron schedule. As an example lets assumme my nifi flow will execute a SQL DELETE and INSERT statement for two groups of tables on two schedule times. 

The simplest implementation will have two generateFlowFileProcessors running on the different schedules, 

    Nifi Cron Expression -> Group
    0 0/15 * * * * ? -> 15min tables schedule
    0 0/30 * * * * ? -> 30min tables schedule 

which load the list tables from a json file which then split to result in each table getting reloaded.

    {
    "tablesToLoad":
        [
            {
            "sourceTable": "TableA",
            "targetTable": "TableB",
            },
            {
            "sourceTable": "TableC",
            "targetTable": "TableD",
            }
        ]
    }

My problem is this doesn't scale well, if need to add other schedules and more table groups. I need to add more generateFlowFileProcessors and schedules to the flow for each reconfiguration.

My preference is to have a single generateFlowFileProcessor which runs on a 5min schedule. The exact schedule that each table runs will be defined as part of the externally loaded config json blob

    {
    "tablesToLoad":
        [
            {
            "sourceTable": "TableA",
            "targetTable": "TableB",
            "schedule": "0 0/15 * * * * ?"
            },
            {
            "sourceTable": "TableC",
            "targetTable": "TableD",
            "schedule": "0 0/30 * * * * ?"
            },
            {
            "sourceTable": "TableC",
            "targetTable": "TableD",
            "schedule": "0 0 0/5 * * * ?"
            }
        ]
    }

Given that the flowFile will be generated with a currentTimeStamp, is there a simple or recommended way in which I could evalue the $schedule expression above with the currentTimestamp to determine if the cron expression matches or not?
Don't have an account?
Coming from Hortonworks? Activate your account here