Member since
07-29-2020
196
Posts
44
Kudos Received
48
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
92 | 01-31-2023 10:16 AM | |
144 | 01-25-2023 12:42 PM | |
83 | 01-18-2023 12:26 PM | |
235 | 01-17-2023 10:41 AM | |
70 | 01-13-2023 06:53 AM |
11-08-2022
12:44 PM
Hi, I'm only able to do that via two processors: 1- JotlTransfomJSON : this is to add desired attributes (attr_id & attr_name) to the flowfile json using the following spec: [
{
"operation": "shift",
"spec": {
"#${attr_id}":"attr_id",
"#${attr_name}":"attr_name",
"model": "model",
"cars_drivers": {
"*": {
"id": "cars_drivers[#2].id",
"name": "cars_drivers[#2].name",
"is_processed": "cars_drivers[#2].is_processed"
}
}
}
}
] 2 - UpdateRecord Processor: Once the attributes are added to the Json , you can update those records Is_processed value only when the id matches the attr_id and name matches the attr_name. To do that set the following properties: a- Replacement Value Strategy: Literal Value b- /cars_drivers[*]/id[.=../../../attr_id]/../name[.=../../../attr_name]/../is_processed : 1 Hope that helps, if it does please accept solution.
... View more
- Tags:
- updaterecord
10-20-2022
09:52 AM
Hi , Can you provide examples of the different json input?
... View more
10-16-2022
08:24 AM
Hi, I think you were close with the last option but did not use the correct syntax with null values where instead of '<> null', use 'is not null' as follows: select * from FLOWFILE where RPATH(fields, '/field') is not null However when I run that I got the correct result but for some reason a MapRecord Syntax is added to each array element : [ {
"fields" : [ "MapRecord[{field=field1}]", "MapRecord[{field=field1}]" ]
}, {
"fields" : [ "MapRecord[{field=field2}]", "MapRecord[{field=field2}]" ]
}, {
"fields" : [ "MapRecord[{field=field3}]", "MapRecord[{field=field3}]" ]
} ] Not sure why this happens but that should get you close to what you are trying to accomplish. Hope that helps. If it does, please accept solution. Thanks
... View more
10-13-2022
06:23 AM
Hi, Try the following Jolt spec: [
{
"operation": "shift",
"spec": {
"tags": {
"*": {
"tag": "tags.[#2].Parameter",
"value": "tags.[#2].value"
}
}
}
}
] If you find this helpful please accept solution.
... View more
10-13-2022
06:11 AM
Hi, I dont think there is a way to do that in one processor, check out this post around the problem with possible solution: https://community.cloudera.com/t5/Support-Questions/Unable-to-return-a-scalar-value-for-the-expression-from-NIFI/td-p/213677 If you find this helpful please accept solution.
... View more
09-28-2022
05:10 PM
Hi, Based on the link below , it doesnt seem like you can access the variable registry directly from an ExecuteScript processor. You can either use the API - as the referenced article in the link below- to retrieve the value or just use an UpdateAttribute processor to get the value of the variable using Expression Language and store into flowfile attribute before passing it to the groovy script where its easy to retrieve the incoming flowfile attribute. https://community.cloudera.com/t5/Support-Questions/NiFi-Is-it-possible-to-access-Processor-Group-variables/m-p/204283 If that helps please accept solution. Thanks
... View more
09-28-2022
06:13 AM
Hi, I dont think it matters which system the path is located as long you have access to it. Do you have access to the path \\10.1.1.35\nifi from the machine where nifi is deployed? I'm not Linux expert but you need to test if the path with the file name is accessible by doing some system command or trying to access via file explorer.
... View more
09-27-2022
09:16 AM
1 Kudo
Hi, Is it always the case that when you have one address you will have a map and if its multiple addresses then its an array? If that is true, then can use the EvaluateJsonPath processor to get the size of child elements of the "addresses" key with the following json path expression: $.addresses.length() If you have one address then the result is 1 otherwise >1. Then you can use RouteOnAttribute when the length is 1 to Jolt Transformation processor with the following spec to convert to Array: [
{
"operation": "shift",
"spec": {
"addresses": {
"*": "addresses.[].&"
},
"*": "&"
}
}
] If that helps please accept solution. Thanks
... View more
09-26-2022
07:23 AM
1 Kudo
Hi, You dont have to use the ExtractText processor for this. Use the EvaluateJsonPath processor with the following configuration: If you find this helpful please accept solution. Thanks
... View more
09-25-2022
06:58 AM
Hi, Have you looked into the ControlRate processor. Its not exactly doing what you want but it allows you to put a time to hold a flowfile before it gets released to the next processor. For example if you think that you need a second to process one flowfile from the moment its fetched to the moment its processed by PutKudu, then you can configure the ControlRate as follows:
... View more
09-23-2022
10:51 AM
I dont think there is an out of the box processor where you can utilize such thing. However you can do some workaround where you can use the ExecuteSQL processor instead since this processor allows you to return the stored proc output in Avro format in new flowfile based on whatever your select statement is in the ExecuteSQL SQL Select Query property. Since this will generate new flowfile, the assumption here is that you dont care about the original flowfile. before going farther and give you an example how to do it, do you want to preserve the original flow file and you were thinking of adding the stored proc output as attribute?
... View more
09-22-2022
03:06 PM
Hi, This question has been asked before several times as you can see in the links below. your best option is to use an ExecuteScript processor and add the output as an attribute to the incoming flowfile. https://stackoverflow.com/questions/72348844/get-output-of-stored-procedure-using-putsql-in-nifi https://stackoverflow.com/questions/57359497/in-nifi-groovy-script-how-to-execute-stored-procedure http://funnifi.blogspot.com/2016/04/sql-in-nifi-with-executescript.html Hope that helps.
... View more
09-20-2022
10:21 AM
Hi, Please try the following spec: [
{
"operation": "shift",
"spec": {
"timestamp": {
"*": {
"@(2,resourceid)": "[&1].resourceid",
"@": "[&1].timestamp"
}
},
"Key": {
"*": {
"@": "[&1].key"
}
},
"data": {
"*": {
"@": "[&1].data"
}
}
}
}
] If you find this helpful, please accept solution. Thanks
... View more
09-19-2022
07:22 AM
Hi, I dont think the following formula is correct: "end": "=${end:toDate('yyyy-MM-dd HH:mm:ss'):toNumber():plus(900000):format('yyyy-MM-dd HH:mm:ss')}" You are trying to use Expression Language in the Jolt Expression and that is not going to work. Also the expression ${end...} assumes that you have an attribute called "end" which I don think its the case. I think what you need to do is the following (assuming you will need to split each json record into its own flowfile): 1- SplitJson 2- Use EvaluateJsonPath to extract the "end" date as an attribute 3- Use JoltTransformationJson where you utilize the attribute from above to do the addition as follows: [
{
"operation": "modify-overwrite-beta",
"spec": {
"end": "${end_attribute:toDate('yyyy-MM-dd HH:mm:ss'):toNumber():plus(900000):format('yyyy-MM-dd HH:mm:ss')}"
}
}
] Hope that helps, if it does please accept solution.
... View more
09-19-2022
06:58 AM
Hi, Is your input in Json format? if its json do you need to completely replace it with the output you specified? If that is the case then you dont need Jolt transformation, instead you can do the following: 1- Place the Gallery_Ids attribute as flowfile content. You can do that using the ReplaceText processor where the Replacement Value is the attribute ${GALLERY_IDS} 2- Do SplitJson to split the Array into a different flow file, this should give 7 flowfiles with the values: 1,2,3,4,5,6 & 7 3- Add another ReplaceText processor that will capture each split from above and replace the flowfile content with the following template in the Replacement Value: {"GALLERY_ID":"$1","PERSON_ID":"test$1"} Leave the Search Value as (?s)(^.*$) This should again give you 7 flowfiles with the expected output format. Hope that helps, if it does please accept solution. Thanks
... View more
09-19-2022
06:11 AM
1 Kudo
hmmm, you are right! the ValidateXML processor only accepts variables from the variable registry as noted in the Schema File property description. @MattWho , @araujo do you know why is this the case for the ValidateXML and what would be the proper solution? The only way I can think of in this case is updating the variable in the variable registry through the API to assign the proper value to the schema.name variable: https://stackoverflow.cm/questions/52010827/how-to-change-nifi-variable-registry-using-rest-api
... View more
09-18-2022
01:43 PM
Hi, To answer your question, it depends: 1- If the attribute value that on the xml flowfile can be used to derive the XSD file name\path then you can use Expression Language to construct the XSD file name from the attribute value when assigning the "Schema File" property. For example if all XSD have the following filename format [SomeID]_XSD_File.xsd and the attribute "Id_Attribute" has the ID then the "Schema File" property can be set to: ${Id_attribute:append('_XSD_File.xsd ')} 2- If the xsd file name cant be derived through the attribute value and you have to do some If-Else conditions based on the attribute then you can use ExecuteScript processor do that and set new flowfile attribute with the xsd file schema path. To learn how to use the ExecuteScript to add new attribute based on some custom code you can check the "Recipe: Add an attribute to a flow file" under: https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-1/ta-p/248922 3- If you dont want to use ExecuteScript processor, you can utilize the UpdateAttribute Processor by creating different Rules, Conditions & Actions to set a File Schema attribute that can be used in the ValidateXML File Schema property. To learn how you can create different rules , condition & action please check the following tutorial: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.5.0/org.apache.nifi.processors.attributes.UpdateAttribute/additionalDetails.html Hope that helps, if it does please accept solution. Thanks
... View more
09-18-2022
10:58 AM
Hi, It doesnt seem like "Stuff" function is recognized function through the QueryRecord sql. I think you have two options: 1- If you are dumping this data into a database where you can use the stuff function there then delegate this to SQL before storing\processing the data. 2- Instead of trying to use QueryRecord processor I would try and use the JoltTransformJson with the following spec: [
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"tempStart": "=split('', @(1,start_time))",
"tempEnd": "=split('', @(1,start_end))",
"start_time": "=concat(@(1,tempStart[0]),@(1,tempStart[1]),':',@(1,tempStart[2]),@(1,tempStart[3]))",
"start_end": "=concat(@(1,tempEnd[0]),@(1,tempEnd[1]),':',@(1,tempEnd[2]),@(1,tempEnd[3]))"
}
}
},
{
"operation": "remove",
"spec": {
"*": {
"temp*": ""
}
}
}
] Not sure how this will perform with large dataset but its worth testing. Hope that helps, if it does please accept solution. Thanks
... View more
09-12-2022
01:05 PM
Hi , Not sure if there is better way, but in my case I was able to get the result using two QueryRecord Processors: 1- Query everything with new field (lets call max_Date) to assign the max between "situation_date" and "new_appointment_date" on each record, in this case the query will be like this: SELECT *, case when situation_date> new_appointment_date then situation_date else new_appointment_date end maxDate
FROM FLOWFILE
2- Next QueryRecord will basically capture the max based on the max_date above from each flowfile records, as follows: SELECT *
FROM FLOWFILE
WHERE max_date = (
SELECT MAX(max_date) from FLOWFILE
) Not sure if this can be done in more efficient way, If anyone can think of better way please advise, otherwise if this works for you please accept solution. Thanks
... View more
09-12-2022
09:04 AM
Hi, I'm not sure if you can use Regex in jolt transformation, but you can use wildcard (*) instead. Keep in mind even with Regex you still need to find the common denominator for each attribute to be able to match, so wildcard should suffice. To use the wildcard you can apply something like this to your spec: [
{
"operation": "shift",
"spec": {
"my_id": "my_id",
"my*company*sa": "my_company_sa",
"*RH": "my_company_rh"
}
}
] Hope that helps, if it does please accept solution. Thanks
... View more
09-12-2022
08:49 AM
Hi, To help you I think we need to understand your question better, so you have a json input like you specified above, from what I understand there are two parts you are trying to solve: 1- You need to process this json if it meets certain condition, in this case the condition is if the Json attribute "dataSourceName" is equal to "Standard CPU Utilization Network Cisco". 2- If the json meets the condition above then you want to apply Json Jolt Transformation to produce the output you specified, is this correct? To answer the first part, I'm not sure if you can have conditional json jolt so you can do the first and the second part in one processor. I think what you need to do is use EvaluateJsonPath processor to extract the value that you need -dataSourceName- into a flow file attribute then use RouteOnAttribute to check the value so that if meets the condition (="Standard CPU Utilization Network Cisco" ) the match relationship will direct the flow file to the Jolt transformation processor to apply needed transformation , otherwise the flowfile will be dropped or direct to unmatched relationship for farther processing. For the second part, its not clear if you need the exact output you specified no matter what the other values in the input json are , which in this case you can use the ReplaceText processor just to replace the whole input json with the expected out, Or you actually need json jolt transformation to capture different values for each json input which in this case its not clear how you are selecting your values, for example on what basis you decided that the value of "Datapoints" is "MemoryFree", also how the attribute "Datas" has the value of "Values[3]". So you need to clarify that better so we can help you with creating the proper jolt spec. Hope that helps. Thanks
... View more
09-12-2022
07:38 AM
Hi, I dont think you need the asterisk "*" in the spec, since you need it on the first level and you dont need to reference any other fields. Just have your spec as follows: [
{
"operation": "modify-default-beta",
"spec": {
"target.organisation": "${target.organisation}"
}
}
]
... View more
09-09-2022
11:01 AM
1 Kudo
Hi, Not sure how you get the URL but if its hard coded in the InvokeHttp Remote URL property , then you can use expression language to set up your parameters as follows: ${literal('https://api.aa/reports/api/order_report/?format=json&dateFrom=#DATEFROM&dateTo=#DATETO'):
replace('#DATEFROM',${now():format('yyyy-MM-dd')}):
replace('#DATETO',${now():toNumber():minus(432000000):format("yyyy-MM-dd")})} Notice: - you can set the date format as desired, in my case I assumed that its in yyyy-MM-dd format. - To subtract 5 days from the current date you have to convert the date to number , then subtract 5 days in milliseconds (1day = 86,400,000 mls ). Hope that helps, if it does please accept solution. Thanks
... View more
09-09-2022
10:38 AM
2 Kudos
Hi , Do you usually have the ID sorrounded with the same phrases all the time? like this: /organisations/ID/detailedLaborcosts If that is the case you can use the following regex in the extract text processor: /organisations/([0-9a-z]+)/detailedLaborcosts Hope that helps.
... View more
09-06-2022
06:55 AM
Hi, you can use the flowfile size attribute "fileSize" to check for the size. for example if you want to process flowfile from S3 that has contents you can create new property in the RouteOnAttribute processor with the following EL: ${fileSize:gt(0)} If you find this helpful, please accept solution. Thanks
... View more
09-03-2022
01:44 PM
Hi , where is the key,value pair are stored? Are they stored as a flowfile attribute or in the input json? there are processors like RouteOnAttribute in case they are stored as attributes where you can create your condition as dynamic property and provide the logical expression there. If the key\value pair is part of the json you can use Extract Text to Extract the key\value pair as an attribute then use RouteOnAttribute or you can try to use RouteOnContent if that will work? If you find this helpful, please accept solution. Thanks Samer
... View more
09-01-2022
12:05 PM
Hi , you can use QueryRecord Processor. For more information please see: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.QueryRecord/additionalDetails.html If you find this helpful please accept solution. Thanks
... View more
08-31-2022
03:02 PM
1 Kudo
Hi , Not sure if this is possible with out of the box processor. I can think of ReplaceText first to replace different delimiters like (-) or white-space (\s) to common delimiter like (,) however if there is a white space before or after other delimiters like (-) or (,) its not going to work. Another option is to use ExecuteScript processor where you try to read each line (after the header) from the flowfile content and then use string split function and try it with different delimiter, once you get two array elements you construct your new string with the new column header and delimiter and transfer to the success relationship.
... View more
08-31-2022
02:17 PM
Hi , You can use JoltJsonTransform Processor for that. So lets assume we have the following input: [
{
"_id": "5d55246b9ac36012c14b5e7c",
"gtin": "4009602235038",
"articleNumber": "59218-376",
"styleGroupId": "59218",
"colorGroupId": "376",
"brand": {
"_id": "57f75950a56ab89d1973ee36"
}
}
] Add JoltTranformJson processor and set the Jolt Specification property to the following, notice this property allows you to use Expression Language therefore you can reference attributes in the flowfile: [
{
"operation": "modify-default-beta", //for modify-overwrite-beta
"spec": {
"*": {
"target.sku": "${target.sku}"
}
}
}
] This will produce the following Json: [ {
"_id" : "5d55246b9ac36012c14b5e7c",
"gtin" : "4009602235038",
"articleNumber" : "59218-376",
"styleGroupId" : "59218",
"colorGroupId" : "376",
"brand" : {
"_id" : "57f75950a56ab89d1973ee36"
},
"target.sku" : "someValue"
} ] If you find this helpful please accept solution. Thanks
... View more
08-27-2022
04:28 PM
1 Kudo
Hi, You can use the ReplaceText Processor as follows: If you find this helpful please accept solution. Thanks
... View more