Member since
07-29-2020
574
Posts
321
Kudos Received
175
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1168 | 12-20-2024 05:49 AM | |
1364 | 12-19-2024 08:33 PM | |
1214 | 12-19-2024 06:48 AM | |
890 | 12-17-2024 12:56 PM | |
1157 | 12-16-2024 04:38 AM |
09-23-2022
10:51 AM
I dont think there is an out of the box processor where you can utilize such thing. However you can do some workaround where you can use the ExecuteSQL processor instead since this processor allows you to return the stored proc output in Avro format in new flowfile based on whatever your select statement is in the ExecuteSQL SQL Select Query property. Since this will generate new flowfile, the assumption here is that you dont care about the original flowfile. before going farther and give you an example how to do it, do you want to preserve the original flow file and you were thinking of adding the stored proc output as attribute?
... View more
09-22-2022
03:06 PM
Hi, This question has been asked before several times as you can see in the links below. your best option is to use an ExecuteScript processor and add the output as an attribute to the incoming flowfile. https://stackoverflow.com/questions/72348844/get-output-of-stored-procedure-using-putsql-in-nifi https://stackoverflow.com/questions/57359497/in-nifi-groovy-script-how-to-execute-stored-procedure http://funnifi.blogspot.com/2016/04/sql-in-nifi-with-executescript.html Hope that helps.
... View more
09-20-2022
10:21 AM
Hi, Please try the following spec: [
{
"operation": "shift",
"spec": {
"timestamp": {
"*": {
"@(2,resourceid)": "[&1].resourceid",
"@": "[&1].timestamp"
}
},
"Key": {
"*": {
"@": "[&1].key"
}
},
"data": {
"*": {
"@": "[&1].data"
}
}
}
}
] If you find this helpful, please accept solution. Thanks
... View more
09-19-2022
07:22 AM
Hi, I dont think the following formula is correct: "end": "=${end:toDate('yyyy-MM-dd HH:mm:ss'):toNumber():plus(900000):format('yyyy-MM-dd HH:mm:ss')}" You are trying to use Expression Language in the Jolt Expression and that is not going to work. Also the expression ${end...} assumes that you have an attribute called "end" which I don think its the case. I think what you need to do is the following (assuming you will need to split each json record into its own flowfile): 1- SplitJson 2- Use EvaluateJsonPath to extract the "end" date as an attribute 3- Use JoltTransformationJson where you utilize the attribute from above to do the addition as follows: [
{
"operation": "modify-overwrite-beta",
"spec": {
"end": "${end_attribute:toDate('yyyy-MM-dd HH:mm:ss'):toNumber():plus(900000):format('yyyy-MM-dd HH:mm:ss')}"
}
}
] Hope that helps, if it does please accept solution.
... View more
09-19-2022
06:58 AM
Hi, Is your input in Json format? if its json do you need to completely replace it with the output you specified? If that is the case then you dont need Jolt transformation, instead you can do the following: 1- Place the Gallery_Ids attribute as flowfile content. You can do that using the ReplaceText processor where the Replacement Value is the attribute ${GALLERY_IDS} 2- Do SplitJson to split the Array into a different flow file, this should give 7 flowfiles with the values: 1,2,3,4,5,6 & 7 3- Add another ReplaceText processor that will capture each split from above and replace the flowfile content with the following template in the Replacement Value: {"GALLERY_ID":"$1","PERSON_ID":"test$1"} Leave the Search Value as (?s)(^.*$) This should again give you 7 flowfiles with the expected output format. Hope that helps, if it does please accept solution. Thanks
... View more
09-19-2022
06:11 AM
1 Kudo
hmmm, you are right! the ValidateXML processor only accepts variables from the variable registry as noted in the Schema File property description. @MattWho , @araujo do you know why is this the case for the ValidateXML and what would be the proper solution? The only way I can think of in this case is updating the variable in the variable registry through the API to assign the proper value to the schema.name variable: https://stackoverflow.cm/questions/52010827/how-to-change-nifi-variable-registry-using-rest-api
... View more
09-18-2022
01:43 PM
Hi, To answer your question, it depends: 1- If the attribute value that on the xml flowfile can be used to derive the XSD file name\path then you can use Expression Language to construct the XSD file name from the attribute value when assigning the "Schema File" property. For example if all XSD have the following filename format [SomeID]_XSD_File.xsd and the attribute "Id_Attribute" has the ID then the "Schema File" property can be set to: ${Id_attribute:append('_XSD_File.xsd ')} 2- If the xsd file name cant be derived through the attribute value and you have to do some If-Else conditions based on the attribute then you can use ExecuteScript processor do that and set new flowfile attribute with the xsd file schema path. To learn how to use the ExecuteScript to add new attribute based on some custom code you can check the "Recipe: Add an attribute to a flow file" under: https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-1/ta-p/248922 3- If you dont want to use ExecuteScript processor, you can utilize the UpdateAttribute Processor by creating different Rules, Conditions & Actions to set a File Schema attribute that can be used in the ValidateXML File Schema property. To learn how you can create different rules , condition & action please check the following tutorial: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.5.0/org.apache.nifi.processors.attributes.UpdateAttribute/additionalDetails.html Hope that helps, if it does please accept solution. Thanks
... View more
09-18-2022
10:58 AM
Hi, It doesnt seem like "Stuff" function is recognized function through the QueryRecord sql. I think you have two options: 1- If you are dumping this data into a database where you can use the stuff function there then delegate this to SQL before storing\processing the data. 2- Instead of trying to use QueryRecord processor I would try and use the JoltTransformJson with the following spec: [
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tempStart": "=split('', @(1,start_time))",
"tempEnd": "=split('', @(1,start_end))",
"start_time": "=concat(@(1,tempStart[0]),@(1,tempStart[1]),':',@(1,tempStart[2]),@(1,tempStart[3]))",
"start_end": "=concat(@(1,tempEnd[0]),@(1,tempEnd[1]),':',@(1,tempEnd[2]),@(1,tempEnd[3]))"
}
}
},
{
"operation": "remove",
"spec": {
"*": {
"temp*": ""
}
}
}
] Not sure how this will perform with large dataset but its worth testing. Hope that helps, if it does please accept solution. Thanks
... View more
09-12-2022
01:05 PM
Hi , Not sure if there is better way, but in my case I was able to get the result using two QueryRecord Processors: 1- Query everything with new field (lets call max_Date) to assign the max between "situation_date" and "new_appointment_date" on each record, in this case the query will be like this: SELECT *, case when situation_date> new_appointment_date then situation_date else new_appointment_date end maxDate
FROM FLOWFILE
2- Next QueryRecord will basically capture the max based on the max_date above from each flowfile records, as follows: SELECT *
FROM FLOWFILE
WHERE max_date = (
SELECT MAX(max_date) from FLOWFILE
) Not sure if this can be done in more efficient way, If anyone can think of better way please advise, otherwise if this works for you please accept solution. Thanks
... View more
09-12-2022
09:04 AM
Hi, I'm not sure if you can use Regex in jolt transformation, but you can use wildcard (*) instead. Keep in mind even with Regex you still need to find the common denominator for each attribute to be able to match, so wildcard should suffice. To use the wildcard you can apply something like this to your spec: [
{
"operation": "shift",
"spec": {
"my_id": "my_id",
"my*company*sa": "my_company_sa",
"*RH": "my_company_rh"
}
}
] Hope that helps, if it does please accept solution. Thanks
... View more