Evaluate attribute value containing expression language

[#1]

Context:

We are using NiFi to read data from a hive store and generate a csv output file. This output file has to be sent out to different recipients, and each recipient has their own requirement for the file name. Some recipients use static name like "enrollments.csv", but others use more dynamic name like "enrollments_<year>_<month>.csv".

We maintain a sql table to store configuration settings for different recipients and file name is one of the settings stored. In the flow, this value is read from the database and assigned to the "filename" attribute, which eventually gets used by PutFile processor when writing content to the disk.

The flow looks something like: ExecuteSQLRecord (Read file name configuration from Sql) --> EvaluateJsonPath (Assign read file name to filename attribute) --> SelectHiveQL (Read data from Hive) --> ConvertRecord (Convert data in Avro format to Csv format) --> PutFile (Writes csv file to disk, using filename attribute value for the name).

Question:

Does a function like eval exist in NiFi where an attribute value containing Expression Language can be evaluated? We are using Expression Language to define and store dynamic file names in sql store, for instance using "enrollments_${now():format('yyyy')}_${now():format('MM')}.csv" to generate file name of "enrollments_<year>_<month>.csv". Though, it doesn't seem to work as filename attribute gets assigned this value in a literal manner; hence needing an extra step to evaluate this expression.

If this function doesn't exist, can this requirement be handled in a different way?

Any help will be highly appreciated.

Thanks!

[#2]

@Puneet Nagpal

-

Running a NiFi Expression Language evaluator against the returned value from an existing Attribute can become dangerous/complicated.

-

Since an EL expression starts with "${" the evaluator would interpret that string as the start of an EL statement. That means that a value that contains that string would be interpreted as an EL statement and throw an exception. Another example may be where the full EL statement structure exists ${<some string>} which is not intended to be an EL statement. If the evaluator ran against that and the <some string> did not match it would return null and value would get changed.
-

So using the updateAttribute to set a your new filename is going to be best option.

filename  -->   enrollments_${now():format('yyyy_MM')}.csv

-

Thanks.

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

[#3]

@Puneet Nagpal

-

*** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion.

-

I understand what you are looking for, but unfortunately there is no existing function that can be used to accomplish this.

[#4]

@Matt Clarke

It would be great if this feature could possibly be considered as a future addition. I'm new to the community, so not sure about the process to create a new feature request. Though, if you direct me to the process documentation, I'm happy to create one.

Thanks!

[#5]

@Puneet Nagpal

-

You can create an account at https://jira.apache.org/jira/secure/Dashboard.jspa

Then you will be able to "Create" "new feature" and "improvement" Jiras against Apache NiFi.
The more detail and context you provide in your request to better chance it has of getting added.

-

Apache NiFi is open source, so feel free to contribute as well.

-

Thank you,

Matt

[#6]

@Matt Clarke

Thanks for the quick response.

Setting filename attribute directly won't scale for us. There could be hundreds of recipients, each with different file name requirement. We could potentially use IfElse, but it'd be highly error prone to maintain hundred+ conditions. More over we already have a tool that captures configuration settings for different recipients and store it in a Sql table, and keeping the Sql table and filename attribute in sync could become a maintenance nightmare.

I understand running EL evaluator against a string value could get complicated. The approach I had in mind was a simple one like having an eval function, for example ${filename:eval()}. This function would run an EL evaluator on the value of the attribute. If the attribute value contains a string "${" that is not intended to be an EL statement, it could be escaped.

Thanks again!

Puneet

[#7]

@puneetn HI, I'm having the same scenario. I wonder, have you ever come up with a solution?

Thanks,

Rosa