Awesome @MattWho . That is great information. However as you said this is going to help moving forward but what about past information? Is there still a way to search the provenance data outside the search feature which doesnt provide capability to search by custom attribute or content?  My guess is not  based on your answer  but I just wanted to confirm. ... View more

Based on my json spec above: 1- customfield_* with multiple records list   2- customfield_* with single records list:   3- customfield_* with no list:   The  asterisk in customfield_* means anything as long there is a customfield , underscore (_) and whatever comes after.   ... View more

I think the issue here is stating what the problem is vs the actual problem. since you starting posting the issue, I noticed the json input has changed 3 times. Remember as someone said "Asking the right question is half way the path to finding the right solution". Allow  me give you some pointers: 1- Please specify the different json input for all possible scenarios and the expected output\s. 2- If you have compound problem then divide and conqueror. Break the big problem into smaller one where each is isolated to its own input, output and  assertions.. 3- Simplify if you can, for example if you have complex json input and you are only facing issue with particular field or nested object then you dont have to post the whole json , instead you isolate where the issue is and provide that part only or create new one that mimic the same structure. 4- If you have code or some formatted data please use the code block "</>" from the menu item for better visibility and readability. Dont post code\data as screenshot. 5- Use screenshots for when suitable , like showing nifi flow, processor configuration ..etc. 6- Refer to the community guidelines for more info: https://community.cloudera.com/t5/custom/page/page-id/Community_Guidelines   Hope that helps.     ... View more

In my jolt spec its expecting customfield_10616 , so in my spec if you notice in the first shift Im using the expression *_* basically to make more dynamic but at least you have to have an underscore but if you think otherwise you can change it to whatever works for your data, for example you can change it to "customfield" , "customfield_*" , "*" , etc.   ... View more

Hi @saquibsk , The below spec should put you on the right path if it doesnt completely solve your issue:     [ // Assign isArray flag field when there is an array (index starts at 0) { "operation": "shift", "spec": { "id": "INTEGRATION_ID", "*": { "*_*": { "@": "&(1,1)", "0": { "#true": "isArray" } } } } } , //If the input is not an array assign default isArray=false { "operation": "default", "spec": { "isArray": "false" } } , //Depending on with isArray true of false: //Transpose values into DEFECT_ROOT_CAUSE_List array { "operation": "shift", "spec": { "customfield": null, "*": "&", "isArray": { "true": { "@(2,customfield)": { "*": { "value": { "@": "DEFECT_ROOT_CAUSE_List[]" } } } }, "false": { "@(2,customfield)": "DEFECT_ROOT_CAUSE_List[]" } } } } , // Finally, get the last element of the array { "operation": "modify-overwrite-beta", "spec": { "DEFECT_ROOT_CAUSE": "=lastElement(@(1,DEFECT_ROOT_CAUSE_List))" } } ]   If that helps please accept solution. Thanks ... View more

Hi @mohdriyaz , I dont think you can accomplish this using the UpdateRecord processor as the added property has to be valid record path where the fields are specified explicitly. I can think of two options to do this:   1- Using JoltTransformJson:  you can use "modify-overwrite-beta" spec to update a field but since the spec allows expression language and flowfile attribute evaluation you can reference the field name in the spec dynamically. In this case the spec will look like this: [ { "operation": "modify-overwrite-beta", "spec": { "${col_name}":"9999" } } ]   2-Using  ExecuteScript: You can write custom script like groovy, python ...etc.  (depending on what Nifi version you are using) the script will basically read the json content into some dictionary or hashmap structure, read the flowfile attribute "col_name" into a variable and use that variable as a key to the field that you need to change in the json dictionary. Finally write then new content to the flowfile and pass to the success relationship. You can do error handling there as well and send any errors to the failure relationship.  For more info on how to update flowfile content using ExecuteScript processor please refer to the "Reading and writing to/from a flow file" section under this link:  https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-2/ta-p/249018 If you find this is helpful please accept solution. Thanks   ... View more

So how are you determining the last file? Based on what? is a file  created\updated date? or is it based on some sort order? ... View more

Hi @Ytch , The way I was able to get to work is through the following instruction: 1- After you have download the jdbc drive from here: https://learn.microsoft.com/en-us/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server?view=sql-server-ver15 you should be able to find the following dll "mssql-jdbc_auth-8.2.2.x64.dll" under the path: ..\sqljdbc_8.2\enu\auth\x64 2- Copy the dll and place under the JAVA Home path bin folder: .. \Java\jdk-21\bin 3- Adjust the DB Connection URL in the DBCPconnectionpool  controller service by adding and setting "integrated security"  flag to true: jdbc:sqlserver://{SQL SEVER NAME};databaseName={DB NAme};integratedSecurity=true; 4- Restart Nifi If that helps please accept solution. Thanks   ... View more