About linssab

linssab · ‎01-13-2023

Ok, I understand now what you are doing. The problem is that you are trying to merge information from one FlowFile to another, and this can't be done in the way you designed. Two FlowFiles mean two separate processes, they don't communicate with one another. You have to use one single FlowFile for the entire process if you need information from both the JSON and XML. I suggest putting the EvaluateXPath after the UpdateAttributes processor, setting the Destination property to "flowfile-attribute" in a way you will have all the attributes in the same flowfile (titlefromXML, guid, Contact Name, caseno, and docno) to use in the JOLT specification. -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.

linssab · ‎01-13-2023

Hey Willy, I'm glad I was able to help. As for the properties, I'm assuming you meant attributes. The JoltTransformJSON supports Expression Language (EL) with Variable Registry and FlowFile attributes. This means you can access any attribute in your flowfile with EL syntax, as ${attributeName} If your flowfile (that reads the XML file) Content is a JSON, you can extract values from the JSON and turn them into attributes with the EvaluateJsonPath processor as below: Then you can use these attributes in the JoltTransform with the syntax written above. However, if the flowfile content is not a JSON and EvaluateJsonPath does not serve the purpose, you can try extracting the information within the FlowFile and turn them into attributes using the ExtractText processor. Let's say teh XLM FLowFile content is: <property1="FieldValue1"> <caseno="foobar"> <property2="FieldValue2"> You can use the syntax below to extract the caseno field and set it as an attribute in the FlowFile caseno=\s*"(\w+)" -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.

linssab · ‎01-12-2023

You have a few options here. Replace text could do the trick with some regex expression but it can get quite complex. I would suggest you use the JoltTransformJSON processor. The follow JOLT Specification will work for you considering your flowfile have the title, id, and contact-name attributes. Replace accordingly. [ { "operation": "modify-overwrite-beta", "spec": { "parameter": { "Title": "${title}", "ExternalId": { "Id": "${id}" } } } }, { "operation": "modify-overwrite-beta", "spec": { "*": { "UnregisteredContacts": { "[1]": { "ContactName": "${contact-name}" } } } } } ]

linssab · ‎01-12-2023

Idk if it is much of a help, but you can try checking the Jolt Transform quick reference There's this link to test JOLT specifications. You can try starting with something like [{ "operation": "shift", "spec": { "*": "&" } }, { "operation": "default", "spec": { "*": { "id": "&(1,0)" } } }]

linssab · ‎01-12-2023

You can try modifying the Record Reader avro schema to something like this: { "name" : "xxx" , "type" : "string" ] , "default" : "" }

linssab · ‎01-12-2023

Hello, This may be a silly question but I'm in a bit of a loss and have been stuck for a couple days now trying to find a solution. I have a flow design where I have to validate a record against an avro schema and insert that record in a Kudu table. The avro validation schema is: { "type" : "record", "name" : "tbcertificati", "fields" : [ { "name" : "num_caso", "type" : "string" }, { "name" : "num_rch", "type" : "string" }, { "name" : "num_cert", "type" : "string" }, { "name" : "seqno", "type" : "string" }, { "name" : "data_inizio_kudu", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "num_vis", "type" : "double" }, { "name" : "data_compilaz_cert", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "data_acquisiz_cert", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "cod_matr_med", "type" : "double" }, { "name" : "medico_struttura", "type" : "string" }, { "name" : "flag_int_ext", "type" : "string" }, { "name" : "cod_cert", "type" : "string" }, { "name" : "cod_prognosi", "type" : "double" }, { "name" : "data_iniz_prognosi", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "data_fine_prognosi", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "desc_cert", "type" : "string" }, { "name" : "data_ripresa_lav", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "flag_postumi", "type" : "string" }, { "name" : "flag_medico_stru", "type" : "string" }, { "name" : "note", "type" : "string" }, { "name" : "prg", "type" : "double" }, { "name" : "crt_time", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "id_usr_crt", "type" : "string" }, { "name" : "mod_time", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "id_usr_mod", "type" : "string" }, { "name" : "fl_annullato", "type" : "string" }, { "name" : "dm_timestamp", "type" : "string" }, { "name" : "dm_txid", "type" : "string" }, { "name" : "dm_operation_type", "type" : "string" } ] } A record can arrive like this: { "DM_TIMESTAMP" : "1970-01-01 00:00:00.000", "DM_TXID" : "1", "DM_OPERATION_TYPE" : "A", "DM_USER" : "STRING", "NUM_CASO" : "1", "NUM_RCH" : "1", "NUM_CERT" : "1", "NUM_VIS" : "", "DATA_COMPILAZ_CERT" : "1970-01-01 00:00:00", "DATA_ACQUISIZ_CERT" : "", "COD_MATR_MED" : "", "MEDICO_STRUTTURA" : "", "FLAG_INT_EXT" : "A", "COD_CERT" : "A", "COD_PROGNOSI" : "1", "DATA_INIZ_PROGNOSI" : "", "DATA_FINE_PROGNOSI" : "", "DESC_CERT" : "", "DATA_RIPRESA_LAV" : "1970-01-01", "FLAG_POSTUMI" : "", "FLAG_MEDICO_STRU" : "A", "NOTE" : "", "PRG" : "1", "CRT_TIME" : "1970-01-01 00:00:00", "ID_USR_CRT" : "STRING", "MOD_TIME" : "1970-01-01 00:00:00", "ID_USR_MOD" : "STRING", "FL_ANNULLATO" : "A", "SEQNO" : "1", "DATA_INIZIO_KUDU" : 1673274352029 } The putKudu processor is as following: Validation record always fails, and if trying to directly insert the record with the PutKudu processor, I get the error: Failed to write due to Can't set primary key column foo to null : java.land.IllegalArgumentException: Can't set primary key column num_caso to null I have trying capitalizing/decapitalizing the field names, with no success. The field clearly has a value associated to it, I don't understand why it is said to be null. Thanks in advance.

linssab · ‎07-15-2022

You can use regular expressions to filter off the attributes you don't need. Let's say that you don't want the attributes "expires_in" and "ext_expires_in" to go on with your flowfile. In the UpdateRecord processor you can use the following expression to remove them from the flowfile in the "D elete Attributes Expression " field: ^.*expires_in.*$ This will match any attribute that contains the string "expires_in" and remove it from the flowfile. Regards, S.

linssab · ‎07-15-2022

Hey @sayak17. You can do this in a few different ways. In any case, the first thing to do is get the access-token value from the JSON response. You can achieve this with an ExecuteScript processor as suggested in this post: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes) I have made the same script in Python if you want and it's available on github. Now you have all the JSON values as attributes in your flowfile. If you want to get rid of attributes you won't need, you can either modify the script yourself or use an UpdateAttribuite processor. To add the "Bearer" string to your access-token value, you can make use of expression language in the second InvokeHTTP processor. Click on the icon inside the processor configuration window to add the attribute "Authorization". In the value field just add the following: ${access-token:prepend("Bearer")} This will take the value of the attribute access-token and prepend the string "Bearer" to it.

linssab · ‎07-14-2022

Hello, NiFi Version: 1.13.2 I am executing a python script within the Execute Script processor running on all nodes. Sometimes it runs all right, but sometimes it gets stuck and do not get flowfiles from the queue, throwing a NullPointerException. As far as I understood, it creates conflicts with the flowfiles resulting in a few files pending in the queue without the possibility to empty it. When right-clicking the queue (with a few hundred flowfiles in there) and selecting "Empty Queue", I just get the message ("X out of Y flowfiles removed"; where Y > X), or "No flowfiles removed" if attempting for a second time. Then, after few minutes, one cluster node crashes. The node error is the following: 07/14/2022 17:41:46 CEST: Node Status changed from CONNECTING to DISCONNECTED due to Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Flows: Local Fingerprint: e48-a975-21705dd65dfa0e0aca15-0c1c-3c8a-9752-179b7491e6a273ebc2c5-31cb-37df-8dcf-5b955bce3d21FUNNELff90c485-cfc9-3b28-9113-f543ed94606b73ebc2c5-31cb-37df-8dcf-5b955bce3d21PROCESSORNO_VALUEDO_NOT_LOAD_ Cluster Fingerprint: e48-a975-21705dd65dfa0e0aca15-0c1c-3c8a-9752-179b7491e6a273ebc2c5-31cb-37df-8dcf-5b955bce3d21FUNNELf379500b-ae38-1ee5-0000-000021eee75273ebc2c5-31cb-37df-8dcf-5b955bce3d21PROCESSORNO_VALUEDO_NOT_LOAD_ I tried offloading the node and reconnecting it but it didn't work and the same error as above persists. Is this some known issue? When running a script accross all nodes causes flowfile conflicts? (I still didn't test running the same load through this processor with the "primary node" execution option - need to wait all nodes to be back up again) The python script I'm running is here on github. Thanks a lot in advance, S.

linssab · ‎07-07-2022

Hey @DigitalPlumber, Yes, it is "secure". It's a very simple docker image with SINGLE_USER_AUTHENTICATION and a PROPS_KEY. The user log shows me anonymous user, which may be the cause of the problem. However, I fail to understand (and sorry the ignorance with NiFi auth methods) how to properly login as an user. NiFi does not prompt me an username or password when accessing the canvas. 2022-07-07 12:47:01,537 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<anonymous>) GET http://localhost:8443/nifi-api/flow/current-user (source ip: 172.21.0.1) 2022-07-07 12:47:01,541 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for anonymous Just in case I'm adding the docker-compose yaml file link here

linssab · ‎07-07-2022

I am trying to import an *.xml template made with NiFi version 1.13.2 into a 1.14 version client (in docker). When trying to upload the *.xml file with the Upload Template button under the Operate panel, I get the following error message: <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 403 Forbidden</title> </head> <body><h2>HTTP ERROR 403 Forbidden</h2> <table> <tr><th>URI:</th><td>/nifi-api/process-groups/8f8e29ed-0181-1000-d7d0-76467ebfce01/templates/upload</td></tr> <tr><th>STATUS:</th><td>403</td></tr> <tr><th>MESSAGE:</th><td>Forbidden</td></tr> <tr><th>SERVLET:</th><td>jerseySpring</td></tr> </table> <hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.42.v20210604</a><hr/> </body> </html> I have never seen this error before. Any clue of what the problem is?

Online	Offline
Last Visited	‎01-30-2023 05:09 AM

Date Registered	‎04-06-2022 06:49 AM
Last Visited	‎01-30-2023 05:09 AM
Total Messages Posted	11
Total Kudos Received	1

Re: Building JSON object for invoke-http

Re: Building JSON object for invoke-http

Re: Building JSON object for invoke-http

Re: Building JSON object for invoke-http

Re: Jolt transform use keys as value

Re: PutDatabasRecord inserting NULL instead of an ...

NiFi Record validation and putKudu

Re: How to dynamically set the value of a property...

Re: How to dynamically set the value of a property...

NullPointerException when executing script in clus...

Re: Template import error

Template import error