Thanks! This scripting approach looks to be working. I'll be sure to look into the ExecuteScript processor more 🙂 ... View more

My Pleasure . I'm glad you were able to get it with the help of Jolt. I agree jolt is a little intimidating initially, but with practice , trial and error you grow to love it 🙂 . A simpler way to represent your spec: [ { "operation": "shift", "spec": { "store": { "book": { "*": "[]" } } } } ] Feel free to post any jolt question or challenges in the future. ... View more

Hi @jarviszzzz , In your DB Lookup Service , I noticed you are aliasing ID as fs_ID in the Lookup Value Columns! I dont think you do an aliasing there , this property is used to list columns that exist in the target table: "A comma-delimited list of columns in the table that will be returned when the lookup key matches. Note that this may be case-sensitive depending on the database." Can you try to set it to just the ID and see if it works.       ... View more

Hi @Anderosn , From previous posts it seems this is a common behavior for COLB column data type and it doesnt seem like you can avoid it. To extract the json value please refer to the following post: https://community.cloudera.com/t5/Support-Questions/Avro-to-Json-adding-extra-delemeters/m-p/380646#M244113   ... View more

Hi @SandyClouds , I ran into this issue before and after some research I found that when you do the ConvertJsonToSQL nifi assigns timestamp data type (value = 93 in the sql.args.[n].type attribute ). When the PutSQL runs the generated sql statement it will parse the value according to the assigned type and format it accordingly. However for timestamp it expects it to be in the format of  "yyyy-MM-dd HH:mm:ss.SSS" so if you are missing the milliseconds in the original datetime value it will fail with the specified error message. To resolve the issue make sure to assign 000 milliseconds to your datetime value before running the PUTSQL processor. You can do that in the source Json itself before the conversion to SQL or after conversion to SQL using UpdateAttribute, by using the later option you have to know which sql.args.[n].value will have the datetime and do expression language to reformat. If that helps please accept solution. Thanks     ... View more

Hi @ALWOSABY , What is the value of the FF_Content? is it the entire JSON record ? if so - as it appears from the specified path - Why not use the EvaluateJsonPath to get whatever values that are  needed and store as attributes by setting the Destination property to flowfile-attribute. See the following post to learn more: https://community.cloudera.com/t5/Support-Questions/How-to-handle-json-using-EvaluateJsonPath-processor-in-NiFi/m-p/295335 If that helps please accept solution. Thanks ... View more

Thanks Samsal ... View more

Hi @MWM , The reason the order changes when the values are null is because they are added with the default spec after they were removed from the initial shift spec when the values are set to blank. There is no quick fix if you want to maintain the order. I can suggest couple of options: 1- Provide additional shift spec to the end of the above spec to re enforce the desired order. The problem with this is that you have to list all the fields as given in the new structure which tends to be challenging specially when you have a lot of fields and complex structure. 2- Handle the setting of blank values  to null for the desired fields before applying the transformation. For that you can use groovy script to write custom code or take advantage of the UpdateRecord Processor and the powerful nifi record path engine that has a lot of built in functions which can help you do such thing easily. Not only both approaches should be easier than the first option because you dont have to list all fields (only one expected to be blank) but also this will simplify your jolt transformation downstream because you dont need to worry about blank values any more. Im not groovy expert , so I can show you how to do it via the UpdateRecord processor which would look like the following: Basically I listed the path for each of the desired fields in the input json  and set the value to the following record path function: /fieldname[not(isEmpty(/fieldname))] which says give me the value of the given field with the condition that the value is not Empty. the isEmpty function returns true if the value is null or blank. If the condition is not met the returned value will always be null. Make sure to set the Replacement Value Strategy to Record Path Value. If that helps please accept solution. Thanks       ... View more

Oh, I successfully managed to integrate and run NiFi 2.0 with Python on Windows using the method you suggested. Thank you so much! ... View more

Your input means a lot, and I appreciate your assistance @DianaTorres, @SAMSAL and @joseomjr. Thank you all for sharing your knowledge and insights! ... View more