Member since
10-05-2022
20
Posts
6
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 7102 | 10-27-2024 09:53 AM | |
| 1554 | 04-07-2023 12:32 AM |
09-01-2025
05:59 AM
Hello dear support team, I’m experiencing the same issue as the original poster and others in the thread. Could you please assist me in updating the email address associated with my account? Thank you very much !
... View more
08-07-2024
04:08 AM
1 Kudo
Hi @Fredi , Its hard to say what is happening without looking at the data where optionalDict is not empty. You only provided data when its empty. Keep in mind that this is not true Python its actually flavor of it called Jython so its not apple to apple when comparing to python. If I can suggest two alternatives: 1 - Since Jython script is going to be deprecated starting from version 2.0 , then I would recommend using groovy instead . Actually parsing json in groovy is much simpler than Jython. Im not sure what version you are using but there is a dedicated processor for executing groovy script called ExecuteGroovyScript that is probably faster than running traditional ExecuteScritp which you can still use it. The script looks like this based on your input : import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
flowFile = session.get()
if(!flowFile) return
def text = ''
// Cast a closure with an inputStream parameter to InputStreamCallback
session.read(flowFile, {inputStream ->
text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
} as InputStreamCallback)
def jsonSlurper = new JsonSlurper()
def jsonData = jsonSlurper.parseText(text)
if(jsonData.directories[0])
{
session.remove(flowFile)
jsonData.directories.each { d ->
newflowfile = session.create()
newflowfile = session.write(newflowfile, {inputStream, outputStream ->
outputStream.write(JsonOutput.toJson(d).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
newflowfile = session.putAttribute(newflowfile, "setId", jsonData.setId.toString())
newflowfile = session.putAttribute(newflowfile, "setName", jsonData.setName)
newflowfile = session.putAttribute(newflowfile, "absolute.path", d.path)
if(jsonData.optionalDict)
{
newflowfile = session.putAttribute(newflowfile, "value1", jsonData.optionalDict.set_entity_relation.intValue.toString())
newflowfile = session.putAttribute(newflowfile, "value2", jsonData.optionalDict.set_entity_relation.stringValue)
}
session.transfer(newflowfile, REL_SUCCESS)
}
}
else session.transfer(flowfile, REL_FAILURE) I have tried the script for both scenarios and it worked as expected. 2- The other alternative is to use other processors (nifi way) to achieve what you want without executing script (not the nifi way) . The execute processor should be left as last option incase the out of the box processors dont suffice or you looking to improve performance in case the flow gets very complicated and inefficient. For this I would use the following processors: 1- JsonEvaluatePath to extract common attribues: setId, SetName, optionalDict.value 1 & 2..etc. 2-Do JsonSplit or QueryRecords on the directories object: this will produce different flowfiles and each flowfile will have the common attribute. 3- JsonEvaluatePath to extract each directory attributes even though its already part of the flowfile content. Hopefully that helps. If it does please accept the solution. Thanks
... View more
07-30-2024
06:00 AM
Hi Matt, thank you for your quick response. I thought that all processors have a built in counter, so I expected to see some no matter what processors are running.
... View more
08-11-2023
06:05 AM
Ok, thanks for the info. I thought it would have been an elegant solution to be able to use the nifi-api to transfer files directly to a port. I have tried ListenHTTP, which suits my task perfectly.
... View more
05-31-2023
06:20 AM
1 Kudo
@steven-matison Thanks man. It is true, the session.commit() method can be found in the abstract processor class, which is why I did not think of adding it. This helped me a lot! Also I needed to close the Inputstream with IOUtils.closeQuietly(stream_content) Thirdly I had to use the enumerate function for the dictionnary, because it couldn't read the line file = session.putAttribute(file, "list_value", d[file]) So I just filled the dict with empty values and used session.putAttribute(file, "list_value", json_data['list'][i]) It is ugly, but works at least.
... View more
04-07-2023
12:32 AM
2 Kudos
Just for everyones information; There was nothing wrong with the code. It was simply that the docker image I used to run Nifi did not build properly and therefore my modifications weren't taken into account.
... View more
01-12-2023
03:44 AM
Idk if it is much of a help, but you can try checking the Jolt Transform quick reference There's this link to test JOLT specifications. You can try starting with something like [{
"operation": "shift",
"spec": {
"*": "&"
}
}, {
"operation": "default",
"spec": {
"*": {
"id": "&(1,0)"
}
}
}]
... View more
12-01-2022
02:33 AM
Hi, thanks for the details. Unfortunately it is not working. I get an empty array [] as output. I have tried it with extract and split mode. I applied the schema text property as suggested with "NestedKey" and "nestedValue" as name. None gives me an output. Meanwhile I have achieved what I wanted using SplitContent and then again another jolt processor. Of course it would be more elegant if I could make it work with ForkRecord.
... View more
10-21-2022
12:28 PM
@Fredi A screenshot of the configuration of your UpdateAttribute processor including main configuration and configuration in the "Advanced" UI would be very helpful in understanding your setup and issue. Thanks, Matt
... View more
10-08-2022
08:23 AM
1 Kudo
Hey @Fredi , I believe the answer for your problem is the processor UpdateRecord. Update record allows you to directly manipulate the fields in your file content. You add dynamic properties to the processor where the key of the property is /<field> (so in your case, '/api_value'), and in the value of this dynamic property you can write down some logic to determine what value to insert into api_value. In the processor, there is a field called "Replacement Value Strategy", which defines how the value of the property will be read. If you set this to "Record Path Value", it means you can now give a path to a different field in your file (url_value!) - I can't test this right now because I'm not at my office, but I'm not entirely sure whether you can manipulate the result after giving a record path (to extract the api_value from the evaluated url_value). Regardless, I'm just about 100% sure this can be done with two processors - One EvaluateJsonPath to extract the url_value into an attribute, then UpdateRecord that uses the 'Literal Value' replacement strategy - with this strategy, you can just add a property with key '/api_value' and value '${url_value}' (or whatever attribute name you gave to the extracted url_value) and once you can access url_value with the expression language (via ${url_value}) you can use all the available functions to manipulate expression language variables. Here's an article with a couple of examples on UpdateRecord: https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267 (I noticed in the article they used some recordPath related functions like "replaceRegex", so I believe there might be a way to use these and then limit the entire issue to just one UpdateRecord processor! Sadly I'm not too familiar with these myself and this was the first time I've seen them) And here's the expression language documentation: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html You can see there are lots of useful functions to extract your api_value once you have ${url_value} as an attribute variable, for example "substring", "find"/"replace", "ifElse", etc. all of which you can try and use to ensure only the api_value is left in the end. Hope this helps! I'm sure using ReplaceText and possibly JoltTransform could provide alternate solutions to the issue, however I believe UpdateRecord is the cleanest solution for this and truly makes use of the processor's abilities. If you struggle to use it correctly, you can reply with an example json and expected output and I'll try to write down the flow when I have time.
... View more