About linssab

linssab · ‎04-12-2023

Ok, now that makes sense. I downloaded from the official repository Index of /dist/nifi (apache.org) and I'm using version 1.13.2 (due to some project requirements)

linssab · ‎04-12-2023

Caused by: java.lang.Exception: The specified login identity provider class 'org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider' is not known to this nifi. I am trying to set up a nifi instance in a Ubuntu Server 22.04 machine. It's been hours I've been tinkering with the configurations and xml files and still I'm getting the error above. These are the configurations I am passing in the nifi.properties, authorizers.xml, login-identity-providers.xml and authorizations.xml files are in the screenshot below. I have tried specifying a user and password in the identity providers file but still nothing...

linssab · ‎01-13-2023

Ok, I understand now what you are doing. The problem is that you are trying to merge information from one FlowFile to another, and this can't be done in the way you designed. Two FlowFiles mean two separate processes, they don't communicate with one another. You have to use one single FlowFile for the entire process if you need information from both the JSON and XML. I suggest putting the EvaluateXPath after the UpdateAttributes processor, setting the Destination property to "flowfile-attribute" in a way you will have all the attributes in the same flowfile (titlefromXML, guid, Contact Name, caseno, and docno) to use in the JOLT specification. -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.

linssab · ‎01-13-2023

Hey Willy, I'm glad I was able to help. As for the properties, I'm assuming you meant attributes. The JoltTransformJSON supports Expression Language (EL) with Variable Registry and FlowFile attributes. This means you can access any attribute in your flowfile with EL syntax, as ${attributeName} If your flowfile (that reads the XML file) Content is a JSON, you can extract values from the JSON and turn them into attributes with the EvaluateJsonPath processor as below: Then you can use these attributes in the JoltTransform with the syntax written above. However, if the flowfile content is not a JSON and EvaluateJsonPath does not serve the purpose, you can try extracting the information within the FlowFile and turn them into attributes using the ExtractText processor. Let's say teh XLM FLowFile content is: <property1="FieldValue1"> <caseno="foobar"> <property2="FieldValue2"> You can use the syntax below to extract the caseno field and set it as an attribute in the FlowFile caseno=\s*"(\w+)" -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.

linssab · ‎01-12-2023

You have a few options here. Replace text could do the trick with some regex expression but it can get quite complex. I would suggest you use the JoltTransformJSON processor. The follow JOLT Specification will work for you considering your flowfile have the title, id, and contact-name attributes. Replace accordingly. [ { "operation": "modify-overwrite-beta", "spec": { "parameter": { "Title": "${title}", "ExternalId": { "Id": "${id}" } } } }, { "operation": "modify-overwrite-beta", "spec": { "*": { "UnregisteredContacts": { "[1]": { "ContactName": "${contact-name}" } } } } } ]

linssab · ‎01-12-2023

Idk if it is much of a help, but you can try checking the Jolt Transform quick reference There's this link to test JOLT specifications. You can try starting with something like [{ "operation": "shift", "spec": { "*": "&" } }, { "operation": "default", "spec": { "*": { "id": "&(1,0)" } } }]

linssab · ‎01-12-2023

You can try modifying the Record Reader avro schema to something like this: {"name": "xxx", "type": "string"], "default": ""}

linssab · ‎01-12-2023

Hello, This may be a silly question but I'm in a bit of a loss and have been stuck for a couple days now trying to find a solution. I have a flow design where I have to validate a record against an avro schema and insert that record in a Kudu table. The avro validation schema is: { "type" : "record", "name" : "tbcertificati", "fields" : [ { "name" : "num_caso", "type" : "string" }, { "name" : "num_rch", "type" : "string" }, { "name" : "num_cert", "type" : "string" }, { "name" : "seqno", "type" : "string" }, { "name" : "data_inizio_kudu", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "num_vis", "type" : "double" }, { "name" : "data_compilaz_cert", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "data_acquisiz_cert", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "cod_matr_med", "type" : "double" }, { "name" : "medico_struttura", "type" : "string" }, { "name" : "flag_int_ext", "type" : "string" }, { "name" : "cod_cert", "type" : "string" }, { "name" : "cod_prognosi", "type" : "double" }, { "name" : "data_iniz_prognosi", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "data_fine_prognosi", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "desc_cert", "type" : "string" }, { "name" : "data_ripresa_lav", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "flag_postumi", "type" : "string" }, { "name" : "flag_medico_stru", "type" : "string" }, { "name" : "note", "type" : "string" }, { "name" : "prg", "type" : "double" }, { "name" : "crt_time", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "id_usr_crt", "type" : "string" }, { "name" : "mod_time", "type" : { "type":"long", "logicalType":"timestamp-millis"}}, { "name" : "id_usr_mod", "type" : "string" }, { "name" : "fl_annullato", "type" : "string" }, { "name" : "dm_timestamp", "type" : "string" }, { "name" : "dm_txid", "type" : "string" }, { "name" : "dm_operation_type", "type" : "string" } ] } A record can arrive like this: { "DM_TIMESTAMP" : "1970-01-01 00:00:00.000", "DM_TXID" : "1", "DM_OPERATION_TYPE" : "A", "DM_USER" : "STRING", "NUM_CASO" : "1", "NUM_RCH" : "1", "NUM_CERT" : "1", "NUM_VIS" : "", "DATA_COMPILAZ_CERT" : "1970-01-01 00:00:00", "DATA_ACQUISIZ_CERT" : "", "COD_MATR_MED" : "", "MEDICO_STRUTTURA" : "", "FLAG_INT_EXT" : "A", "COD_CERT" : "A", "COD_PROGNOSI" : "1", "DATA_INIZ_PROGNOSI" : "", "DATA_FINE_PROGNOSI" : "", "DESC_CERT" : "", "DATA_RIPRESA_LAV" : "1970-01-01", "FLAG_POSTUMI" : "", "FLAG_MEDICO_STRU" : "A", "NOTE" : "", "PRG" : "1", "CRT_TIME" : "1970-01-01 00:00:00", "ID_USR_CRT" : "STRING", "MOD_TIME" : "1970-01-01 00:00:00", "ID_USR_MOD" : "STRING", "FL_ANNULLATO" : "A", "SEQNO" : "1", "DATA_INIZIO_KUDU" : 1673274352029 } The putKudu processor is as following: Validation record always fails, and if trying to directly insert the record with the PutKudu processor, I get the error: Failed to write due to Can't set primary key column foo to null : java.land.IllegalArgumentException: Can't set primary key column num_caso to null I have trying capitalizing/decapitalizing the field names, with no success. The field clearly has a value associated to it, I don't understand why it is said to be null. Thanks in advance.

linssab · ‎07-15-2022

You can use regular expressions to filter off the attributes you don't need. Let's say that you don't want the attributes "expires_in" and "ext_expires_in" to go on with your flowfile. In the UpdateRecord processor you can use the following expression to remove them from the flowfile in the "Delete Attributes Expression" field: ^.*expires_in.*$ This will match any attribute that contains the string "expires_in" and remove it from the flowfile. Regards, S.

linssab · ‎07-15-2022

Hey @sayak17. You can do this in a few different ways. In any case, the first thing to do is get the access-token value from the JSON response. You can achieve this with an ExecuteScript processor as suggested in this post: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes) I have made the same script in Python if you want and it's available on github. Now you have all the JSON values as attributes in your flowfile. If you want to get rid of attributes you won't need, you can either modify the script yourself or use an UpdateAttribuite processor. To add the "Bearer" string to your access-token value, you can make use of expression language in the second InvokeHTTP processor. Click on the icon inside the processor configuration window to add the attribute "Authorization". In the value field just add the following: ${access-token:prepend("Bearer")} This will take the value of the attribute access-token and prepend the string "Bearer" to it.

Online	Offline
Last Visited	‎10-03-2024 05:02 AM

Member Since	‎04-06-2022 06:49 AM
Last Visited	‎10-03-2024 05:02 AM
Posts	13
Kudos received	1

Cloudera Community

Re: Building JSON object for invoke-http

Re: Issues with Single User authentication NiFi

Issues with Single User authentication NiFi

Re: Building JSON object for invoke-http

Re: Building JSON object for invoke-http

Re: Building JSON object for invoke-http

Re: Jolt transform use keys as value

Re: PutDatabasRecord inserting NULL instead of an ...

NiFi Record validation and putKudu

Re: How to dynamically set the value of a property...

Re: How to dynamically set the value of a property...