I think the issue here is stating what the problem is vs the actual problem. since you starting posting the issue, I noticed the json input has changed 3 times. Remember as someone said "Asking the right question is half way the path to finding the right solution". Allow  me give you some pointers: 1- Please specify the different json input for all possible scenarios and the expected output\s. 2- If you have compound problem then divide and conqueror. Break the big problem into smaller one where each is isolated to its own input, output and  assertions.. 3- Simplify if you can, for example if you have complex json input and you are only facing issue with particular field or nested object then you dont have to post the whole json , instead you isolate where the issue is and provide that part only or create new one that mimic the same structure. 4- If you have code or some formatted data please use the code block "</>" from the menu item for better visibility and readability. Dont post code\data as screenshot. 5- Use screenshots for when suitable , like showing nifi flow, processor configuration ..etc. 6- Refer to the community guidelines for more info: https://community.cloudera.com/t5/custom/page/page-id/Community_Guidelines   Hope that helps.     ... View more

In my jolt spec its expecting customfield_10616 , so in my spec if you notice in the first shift Im using the expression *_* basically to make more dynamic but at least you have to have an underscore but if you think otherwise you can change it to whatever works for your data, for example you can change it to "customfield" , "customfield_*" , "*" , etc.   ... View more

Hi @saquibsk , The below spec should put you on the right path if it doesnt completely solve your issue:     [ // Assign isArray flag field when there is an array (index starts at 0) { "operation": "shift", "spec": { "id": "INTEGRATION_ID", "*": { "*_*": { "@": "&(1,1)", "0": { "#true": "isArray" } } } } } , //If the input is not an array assign default isArray=false { "operation": "default", "spec": { "isArray": "false" } } , //Depending on with isArray true of false: //Transpose values into DEFECT_ROOT_CAUSE_List array { "operation": "shift", "spec": { "customfield": null, "*": "&", "isArray": { "true": { "@(2,customfield)": { "*": { "value": { "@": "DEFECT_ROOT_CAUSE_List[]" } } } }, "false": { "@(2,customfield)": "DEFECT_ROOT_CAUSE_List[]" } } } } , // Finally, get the last element of the array { "operation": "modify-overwrite-beta", "spec": { "DEFECT_ROOT_CAUSE": "=lastElement(@(1,DEFECT_ROOT_CAUSE_List))" } } ]   If that helps please accept solution. Thanks ... View more

Hi @mohdriyaz , I dont think you can accomplish this using the UpdateRecord processor as the added property has to be valid record path where the fields are specified explicitly. I can think of two options to do this:   1- Using JoltTransformJson:  you can use "modify-overwrite-beta" spec to update a field but since the spec allows expression language and flowfile attribute evaluation you can reference the field name in the spec dynamically. In this case the spec will look like this: [ { "operation": "modify-overwrite-beta", "spec": { "${col_name}":"9999" } } ]   2-Using  ExecuteScript: You can write custom script like groovy, python ...etc.  (depending on what Nifi version you are using) the script will basically read the json content into some dictionary or hashmap structure, read the flowfile attribute "col_name" into a variable and use that variable as a key to the field that you need to change in the json dictionary. Finally write then new content to the flowfile and pass to the success relationship. You can do error handling there as well and send any errors to the failure relationship.  For more info on how to update flowfile content using ExecuteScript processor please refer to the "Reading and writing to/from a flow file" section under this link:  https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-2/ta-p/249018 If you find this is helpful please accept solution. Thanks   ... View more

So how are you determining the last file? Based on what? is a file  created\updated date? or is it based on some sort order? ... View more

Hi @Ytch , The way I was able to get to work is through the following instruction: 1- After you have download the jdbc drive from here: https://learn.microsoft.com/en-us/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server?view=sql-server-ver15 you should be able to find the following dll "mssql-jdbc_auth-8.2.2.x64.dll" under the path: ..\sqljdbc_8.2\enu\auth\x64 2- Copy the dll and place under the JAVA Home path bin folder: .. \Java\jdk-21\bin 3- Adjust the DB Connection URL in the DBCPconnectionpool  controller service by adding and setting "integrated security"  flag to true: jdbc:sqlserver://{SQL SEVER NAME};databaseName={DB NAme};integratedSecurity=true; 4- Restart Nifi If that helps please accept solution. Thanks   ... View more

Hi @Kiranq , This a basic case of data enrichment and there are multiple ways how you can handle this. If for example instead of the function you have a table with a unique id then you can use LookupRecord with DatabaseRecordLookupService or even SimpleDatabaseLookupService . All you have to do here is basically specify the path where you want the new value to be inserted as dynamic property. However since you have a function instead of simple table\view , then the second option is to use ForkEnrich\JoinEnrich processors . The nice thing about those processors is that you have multiple strategies how you can join the data together as you can read here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html   Let me know which one works for you better and let me know if you face any issues regarding using either. If you find this is helpful please accept solution Thanks       ... View more

Hi @sajidkhan , Have you actually ran the flowfiles through the ConvertRecord processor? I noticed you have 8 files queued in the ExecuteSQL success relationship. If you open the flowfiles to inspect from there they are going to be in AVRO format since ExecuteSQL will produce such format by defualt. You have to right click the ConvertRecord Processor and select Start to get the files processed through (Keep the Put File processor Stopped)  and then under the ConvertRecord success relationship you should have 8 converted flowfiles that you can inspect to see if they got converted to CSV. If the conversion was done accordingly then you can start the PutFile processor to save them to your local drive. Since you seem to be new to Nifi, I would recommend referring to the following article to understand how Nifi and Nifi processors  work: https://docs.cloudera.com/cfm/2.0.4/nifi-user-guide/topics/nifi-processor_anatomy.html https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html https://nifi.apache.org/docs/nifi-docs/html/user-guide.html https://nifi.apache.org/docs/nifi-docs/html/overview.html   If you find this is helpful please accept solution. Thanks     ... View more