Member since
10-05-2022
19
Posts
4
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
530 | 04-07-2023 12:32 AM |
08-01-2024
07:31 AM
Hello, I have a problem with execute script processor. I have an incoming flowfile with json content like this: {
"setName": "name",
"optionalDict": {},
"directories": [
{
"path":"path/to/file",
"fileName":"filename",
"id":1
}
],
"setId":12
} My script is the following: import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
from org.apache.nifi.processor.io import InputStreamCallback
flowFile = session.get()
if (flowFile != None):
stream_content = session.read(flowFile)
text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8)
IOUtils.closeQuietly(stream_content)
json_data = json.loads(text_content)
session.remove(flowFile)
d = {}
for x in range(len(json_data['directories'])):
d["flowfile{0}".format(x)] = None
for i, file in enumerate(d):
file = session.create()
file = session.putAttribute(file, "setId", str(json_data['setId']))
file = session.putAttribute(file, "setName", json_data["setName"])
file = session.putAttribute(file, "absolute.path", json_data['directories'][i]["path")
if json_data['optionalDict'] is not None:
if bool(json_data['optionalDict']):
file = session.putAttribute(file, "value1", str(json_data['set_entity_relation']['intValue']))
file = session.putAttribute(file, "value2", json_data['set_entity_relation']['stringValue'])
session.transfer(file, REL_SUCCESS)
session.commit() So what I aim to do is to commit one flowfile per entry in "directories" and write some values to the flowfiles attributes. This works fine if the "optionalDict" is not null and has the necessary entries. When it is an empty dict though, the script still tries to write the optional values and fails, since there are none. When I execute the same python code on my local scripts the if clauses prevent this. Why does the executeScript processor ignore the two if clauses? I have tried different ways to stop it from executing the optional steps, but nothing works. Also it keeps telling me the error is on the line of the if clauses, when in reality it is two lines further down. This is the error message: TypeError: putAttribute(): 3rd arg can't be coerced to String Well, it shouldn't try to. What could I try to do?
... View more
Labels:
- Labels:
-
Apache NiFi
07-30-2024
06:00 AM
Hi Matt, thank you for your quick response. I thought that all processors have a built in counter, so I expected to see some no matter what processors are running.
... View more
07-29-2024
03:05 AM
1 Kudo
Hello there, I learned about nifi counters, but when I open my nifi instance and click on the counters button in the main menu, it just opens an empty list. Same when I call nifi-api/counters. It is just an empty list. I do have processors and processgroups running. So does anyone have an explanation why that is? I use nifi 1.23.2 on my local machine. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
08-11-2023
06:05 AM
Ok, thanks for the info. I thought it would have been an elegant solution to be able to use the nifi-api to transfer files directly to a port. I have tried ListenHTTP, which suits my task perfectly.
... View more
08-10-2023
07:14 AM
Hello, how do I properly use the nifi-api to create flowfile from a file? I tried to do what is described here: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html under 'Data transfer' I've tried to use a simple curl command to transfer a file as flowfile to an input port as follows: curl -X POST -F 'file=@Documents/Nifi_Docs/example.txt' http://10.0.1.111:8080/nifi-api/data-transfer/input-ports/dac743ae-0189-1000-c6a6-efd211a162d1/transactions/1010/flow-files As response I get a 415 error Unsupported Media Type. What is the supported media type then? Anyone tried this before? What is the transaction id good for? I just put 1010 as a random number. I also tried sending from postman
... View more
Labels:
- Labels:
-
Apache NiFi
05-31-2023
06:20 AM
1 Kudo
@steven-matison Thanks man. It is true, the session.commit() method can be found in the abstract processor class, which is why I did not think of adding it. This helped me a lot! Also I needed to close the Inputstream with IOUtils.closeQuietly(stream_content) Thirdly I had to use the enumerate function for the dictionnary, because it couldn't read the line file = session.putAttribute(file, "list_value", d[file]) So I just filled the dict with empty values and used session.putAttribute(file, "list_value", json_data['list'][i]) It is ugly, but works at least.
... View more
05-31-2023
03:26 AM
Hi @cotopaul, thanks for your remark. I have seen the cookbooks, but there don't seem to be examples for creating multiple flowfiles from one incoming flowfile. As you can see from my edit I have advanced a bit, Also, I don't even want to generate content (I do that later with other processors). I just want to create flowfiles with some custom attributes with different values.
... View more
05-31-2023
12:18 AM
Hi, I am trying ro read from a json file and create flowfiles with the kews as attributes and one flowfile per value in the list. The json file looks something like this: { "key1": null, "key2": null, "list": ["foo", "bar"], "key3": 1 } This is my execute script: import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
from org.apache.nifi.processor.io import InputStreamCallback
flowFile = session.get()
if (flowFile != None):
stream_content = session.read(flowFile)
text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8)
json_data = json.loads(text_content)
for i in json_data["list"]:
flowFile2 = session.create()
flowFile2 = session.putAttribute(flowFile2, "key3", str(json_data["key3"]))
flowFile2 = session.putAttribute(flowFile2, "key1", json_data["key1"])
flowFile2 = session.putAttribute(flowFile2, "list_value", i)
session.transfer(flowFile2, REL_SUCCESS)
session.remove(flowFile) Now I keep getting an error that I need to close the process session first because my original flowfile is still in use. How can I iterate over the list to create multiple flowfiles properly? I couldn't find any similar case or example of this sort. Thanks for tips and help. Edit: So I changed my code to create a dictionnary in order to use a new variable name for each flowfile created, but now I keep getting a key error when I want to transfer them to the success relationship. import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
from org.apache.nifi.processor.io import InputStreamCallback
flowFile = session.get()
if (flowFile != None):
stream_content = session.read(flowFile)
text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8)
json_data = json.loads(text_content)
d = {}
for x in range(len(json_data["list"])):
d["flowfile{0}".format(x)] = json_data['list'][x]
for file in d:
file = session.create()
file = session.putAttribute(file, "key3", str(json_data["key3"]))
file = session.putAttribute(file, "key2", json_data["key2"])
file = session.putAttribute(file, "list_value", d[file])
session.transfer(file, REL_SUCCESS)
session.remove(flowFile)
... View more
Labels:
- Labels:
-
Apache NiFi
04-07-2023
01:25 AM
Hello, is there such a thing as an escape character? I would need keynames with a "." in it. In this example something like: Output { "id": "3240", "files" : { "mac.File": "mac-20200806.json", "window.File": "window-20200806.json", "linux.File": "linux-20200806.json" } } But dots are interpreted as nesting objects. Thanks Edit: Nevermind, I found we can use escape characters with "\\"
... View more
04-07-2023
12:32 AM
2 Kudos
Just for everyones information; There was nothing wrong with the code. It was simply that the docker image I used to run Nifi did not build properly and therefore my modifications weren't taken into account.
... View more