Dynamically Accessing Parameter Contexts Values

7 replies
8964 views
Translate All

I am storing several JSON Schemas as Parameter Contexts, and I need to access the schemas dynamically to validate flowfiles. But which schema depends on the flowfile, so I want to access a particular schema in the Parameter Contexts dynamically using an attribute value. How do I do that?

Here is what I've tried:

#{${schema.name}}

${schema.name:evaluateELString()}

The second technique I am using creates the exact string that I need #{Blue_JSON_Schema}, and then I try to perform an eval on it. But unfortunately it doesn't work either.

Hopefully someone knows how to do this. Thanks for taking the time to read this.

Apache NiFi

Amnr

Silver ⭐️

I am having the same problem with version 1.20.0. I am using the advanced properties from "UpdateAttribute" to set up the attribute name to:

s3_bucket #{s3_my_bucket}

and then using it on my next processor:

s3_buc: ${s3_bucket:evaluateELString()}

And is not working, the value that I am getting is:

s3_buc Empty string set

This functionality used to work in prev. versions.

Thanks

Translate

MattWho

Titanium ⭐️⭐️⭐️

@ChuckE

I would not expect this to work. The "evaluateELString()" NiFi Expression Language (NEL) function triggers the evaluation of any NEL statements found in the subject passed to that function. An NEL statement always starts with "${"; however, what you have is "#{" since you have a parameter context reference in your subject. So the expected output would be the literal subject in your case.

Somewhere in your dataflow you are assigning the literal parameter name to a FlowFile Attribute. Why not evaluate the parameter rather then assign it as a literal string in an FlowFile attribute? Perhaps some more context/details around your use case would help here?

Thanks,

Matt

Translate

ChuckEAuthor

Gold ⭐️

The concept is EXACTLY the same concept as using the AvroSchemaRegistry, which holds a bunch of schemas, and as each flowfile gets read by a single processor that uses a NiFi Recordset Reader/Writer service (e.g. ConvertRecord), they will individually reference their own schema. Some are Red data, some are Blue data, and some are Purple data, and they each dynamically reference their corresponding Avro schema. In fact, you can specify the "schema.name" attribute, which references said schema in the AvroSchemaRegistry.

The premise of my inquiry is that I want to achieve something similar using JSON Schemas despite the fact there is no JSON Schema Registry. So I want to use Parameter Context parameters to hold the schemas, then dynamically refer to them in a similar manner as we can do with Avro schemas.

Translate

MattWho

Titanium ⭐️⭐️⭐️

@ChuckE
Understood. What i am curious about is where in your dataflow do you implement the logic to determine which schema needs to be used?

So you have some FlowFile with json content in your dataflow.
- You then need to determine which schema needs to go with this specific FlowFile's content.
- Then you want to return that schema text from a parameter context.

So how do you make that determination of what schema goes with which FlowFile? Simply by where data was consumed from?

Have you considered using the "Advanced" UI of the updateAttribute to create rules based on how you make your determinations to add a new FlowFIle Attribute with the extracted schema from the parameter context(s)?

Thanks,

Matt

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.