Hi @MWM , The reason the order changes when the values are null is because they are added with the default spec after they were removed from the initial shift spec when the values are set to blank. There is no quick fix if you want to maintain the order. I can suggest couple of options: 1- Provide additional shift spec to the end of the above spec to re enforce the desired order. The problem with this is that you have to list all the fields as given in the new structure which tends to be challenging specially when you have a lot of fields and complex structure. 2- Handle the setting of blank values  to null for the desired fields before applying the transformation. For that you can use groovy script to write custom code or take advantage of the UpdateRecord Processor and the powerful nifi record path engine that has a lot of built in functions which can help you do such thing easily. Not only both approaches should be easier than the first option because you dont have to list all fields (only one expected to be blank) but also this will simplify your jolt transformation downstream because you dont need to worry about blank values any more. Im not groovy expert , so I can show you how to do it via the UpdateRecord processor which would look like the following: Basically I listed the path for each of the desired fields in the input json  and set the value to the following record path function: /fieldname[not(isEmpty(/fieldname))] which says give me the value of the given field with the condition that the value is not Empty. the isEmpty function returns true if the value is null or blank. If the condition is not met the returned value will always be null. Make sure to set the Replacement Value Strategy to Record Path Value. If that helps please accept solution. Thanks       ... View more

Hi @MWM , The following worked for me: The GenerateFlwoFile has json content and an attribute flowfile_id with value 123 In the ReplaceText I replace everything with empty string: In the UpdateAttribute Im adding new Attribute new_attr with value of 555 The MergeConent is configured as follows: I'm using flowfile_id as Correlation Attribute Name. Also notice how I set the "Minimum Number of Entries" to 2 so that original flowfile will wait until the second  ready. The result is the original flowfile content with the new added attribute.   Another option to MergeRecord is you can use PutDistributedMapCache and FetchDistributedMapCache to store the original content into Cache, then do whatever is needed to get the new attributes  and finally fetch the original content again , this will give you the original flowfile including the new attributes. The only caveat with this approach that you have to create two controller services:  DistributedMapCacheClientService & DistributedMapCacheServer.  Another issue with this DistributedMapCacheClientService  is that you have to provide a server hostname which could be the same as your nifi node, however this produces a single point of failure specially when you have cluster. For more info: https://stackoverflow.com/questions/44590296/how-does-one-setup-a-distributed-map-cache-for-nifi If that helps please accept solution. Thanks       ... View more

Hi @SAMSAL your solution is workink, thank you so much! ... View more

Is the problem that the other system doesnt take negative values? ... View more

@SAMSAL Thank you for help, everything works unfortunately except the date, but as you said, it is more complicated   ... View more

@MWM @cotopaul   If you get the record reader/writer using the schema(s) you want,  you do not have to do any magic to convert values, it should just work.     Only use, inferSchema long enough to get the structure when you have none.  Then copy/paste it and use it as @cotopaul has described in place of InferSchema.  You can also use Schema Registry.  Make the edits you need to satisfy reader (upstream), writer (downstream) as they are sometimes needing minor adjustments like in this case. ... View more

@MWM Before sendEmail you need to add a DetectDuplicate processor.    https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.DetectDuplicate/   You can find a sample template here: https://github.com/steven-matison/NiFi-Templates/blob/master/DetectDuplicate_DistributedMapCache_Demo.xml   ... View more

Hi @MWM ,   What you are describing is a classical data enrichment pattern that can be achieved using ForkEnrichment & JoinEnrichment processors. For more information on this please refer to : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html   Based on your scenario the "SQL" strategy of the JoinEnrichment will work the best for you since you can select the main fields (system, name, surname, phone, mail) from the original flowfile data and select proffesion and department from enrichment result:   SELECT o.system, o.name, e.surname,o.phone, o.mail, e.profession, e.department FROM original o LEFT OUTER JOIN enrichment e ON o.name= e.name Since you are splitting the CSV and enrich per record then you can just join by name. If you have an API where you can get a collection of user information then you dont have to split and you can do the enrichment on multiple records from the CSV vs. returned records from the API json output , however be aware that if you have large data set this strategy "... can lead to heap exhaustion and cause stability problems or OutOfMemoryErrors to occur". Please review the link above to see how this can be mitigated.   If you find this is helpful please accept solution. Thanks     ... View more

@MWM  Your description is not super clear to me. FlowFile Attributes are in format of <attribute name>=<attribute value> example key=value FlowFile attributes: Attribute name = Attribute value name=toyota type=car Here is how I interpret what you have provided: - You have a FlowFile that has a custom FlowFile "Attribute Name" on it "name". - That "name" Attribute has no value associated to it. (name = ) The UpdateAttribute processor allows you to add new key=value (attribute name = attribute value) attribute pairs to an existing FlowFile or modify the value of an existing FlowFile Attribute Name. You can NOT create a new attribute and then modify or use that new attribute within the same processor in which it was created. Lets say you have a FlowFile with the example FlowFile attributes and values I shared above. Then you use the UpdateAttribute processor to change the value on the Attribute "name" to all uppercase. The NiFi Expression Language (NEL) statement you created: ${name:toUpper()} will take the subject "name" and return it's value "toyota".  That returned value is passed to the toUpper() NEL function which converts the it to Uppercase "TOYOTA".  The resulting value is then written to the FlowFile attribute defined in the UpdateAttribute processor. Assuming an UpdateAttribute processor configured as below: The NEL expression resulting a new value of "TOYOTA" would be written to the FlowFile Attribute "name" since "name" already exists as an attribute on the FlowFile, its current attribute value of "toyota" would be replaced with new value "TOYOTA". Hope this helps clarify how NiFi FlowFile Attributes and the UpdateAttribute processor is used. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more