Skip to main content
SandyClouds
SandyClouds
Gold ⭐️
August 31, 2023
Solved

How to send a attribute to downstream processor and use it in JoltTransformJSON

I had a flow which needs to be replicated for 100 jobs.

one of my jolt transform looks like this.

 

[{
  "operation": "default",
  "spec": {
    "*": {
      "id": "#{customer_id_1}",
      "warehouse_loaddate" : "${now():format('yyyy-MM-dd HH:mm:ss', 'CET')}"
    }
  }
}]

 

The customer_id_1 is a parameter currently, which is different in all 10 different jobs.  like there are many different parameters in the context like 

customer_id_2

customer_id_3 etc..

With this I need to touch my code at 100 different places...

But since I know the customer ID is present in my upstream processor.. so I just want to name the parameter customer_id and get the value from upstream dynamically... Is there a way that I can send a attribute to downstream processor and use it in jolt ?

Without using parameters from parameter context ? 

We use #{<id>} and ${<func>} to access parameters and Functions.. but how can i refer an attribute from my upstream processor ?

 

Please help me understand this.

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
September 19, 2023

@SandyClouds 
The "Jolt Specification" property in the JoltTransformJson processor supports NiFi Expression Language (will be evaluated using flow file attributes and variable registry).  This means that any attribute present on the FlowFile passed to the JoltTransform processor can be used in the spec.   So if you have an upstream processor adding an attribute to your FlowFile with the attribute name "customer_id", you can simply use NiFi Expression Language in your spec to pull in that attributes value. 

NiFi Parameters are defined using "#{<parameter name>}" which will look up that parameter in the parameter context defined on the NiFi Process Group and return its value.

NiFi Expression language is defined by starting with "${" and ending with "}".  Between that you can bring in an attributes value, apply NiFi Expression Language functions against an attributes value, etc.  Simplest NiFi Expression Language statement would be "${customer_id}", which will look for an attribute in the FlowFile being processed and return its value.  We refer to "customer_id" as the "subject" in that NiFi EL. NiFi EL does have a bunch of subjectless functions like "now()" that you are using, but more typical use cases start with a subject (FlowFile attribute).  I strongly encourage you to read through the linked NiFi EL guide to help you better understand the complexity and many ways you can used NiFi EL statements.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

View in thread

1 reply

MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
September 19, 2023

@SandyClouds 
The "Jolt Specification" property in the JoltTransformJson processor supports NiFi Expression Language (will be evaluated using flow file attributes and variable registry).  This means that any attribute present on the FlowFile passed to the JoltTransform processor can be used in the spec.   So if you have an upstream processor adding an attribute to your FlowFile with the attribute name "customer_id", you can simply use NiFi Expression Language in your spec to pull in that attributes value. 

NiFi Parameters are defined using "#{<parameter name>}" which will look up that parameter in the parameter context defined on the NiFi Process Group and return its value.

NiFi Expression language is defined by starting with "${" and ending with "}".  Between that you can bring in an attributes value, apply NiFi Expression Language functions against an attributes value, etc.  Simplest NiFi Expression Language statement would be "${customer_id}", which will look for an attribute in the FlowFile being processed and return its value.  We refer to "customer_id" as the "subject" in that NiFi EL. NiFi EL does have a bunch of subjectless functions like "now()" that you are using, but more typical use cases start with a subject (FlowFile attribute).  I strongly encourage you to read through the linked NiFi EL guide to help you better understand the complexity and many ways you can used NiFi EL statements.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate
    Terms & ConditionsAccessibility statement