Member since
10-05-2022
16
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
365 | 04-07-2023 12:32 AM |
08-11-2023
06:05 AM
Ok, thanks for the info. I thought it would have been an elegant solution to be able to use the nifi-api to transfer files directly to a port. I have tried ListenHTTP, which suits my task perfectly.
... View more
08-10-2023
07:14 AM
Hello, how do I properly use the nifi-api to create flowfile from a file? I tried to do what is described here: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html under 'Data transfer' I've tried to use a simple curl command to transfer a file as flowfile to an input port as follows: curl -X POST -F 'file=@Documents/Nifi_Docs/example.txt' http://10.0.1.111:8080/nifi-api/data-transfer/input-ports/dac743ae-0189-1000-c6a6-efd211a162d1/transactions/1010/flow-files As response I get a 415 error Unsupported Media Type. What is the supported media type then? Anyone tried this before? What is the transaction id good for? I just put 1010 as a random number. I also tried sending from postman
... View more
Labels:
- Labels:
-
Apache NiFi
05-31-2023
06:20 AM
1 Kudo
@steven-matison Thanks man. It is true, the session.commit() method can be found in the abstract processor class, which is why I did not think of adding it. This helped me a lot! Also I needed to close the Inputstream with IOUtils.closeQuietly(stream_content) Thirdly I had to use the enumerate function for the dictionnary, because it couldn't read the line file = session.putAttribute(file, "list_value", d[file]) So I just filled the dict with empty values and used session.putAttribute(file, "list_value", json_data['list'][i]) It is ugly, but works at least.
... View more
05-31-2023
03:26 AM
Hi @cotopaul, thanks for your remark. I have seen the cookbooks, but there don't seem to be examples for creating multiple flowfiles from one incoming flowfile. As you can see from my edit I have advanced a bit, Also, I don't even want to generate content (I do that later with other processors). I just want to create flowfiles with some custom attributes with different values.
... View more
05-31-2023
12:18 AM
Hi, I am trying ro read from a json file and create flowfiles with the kews as attributes and one flowfile per value in the list. The json file looks something like this: { "key1": null, "key2": null, "list": ["foo", "bar"], "key3": 1 } This is my execute script: import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
from org.apache.nifi.processor.io import InputStreamCallback
flowFile = session.get()
if (flowFile != None):
stream_content = session.read(flowFile)
text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8)
json_data = json.loads(text_content)
for i in json_data["list"]:
flowFile2 = session.create()
flowFile2 = session.putAttribute(flowFile2, "key3", str(json_data["key3"]))
flowFile2 = session.putAttribute(flowFile2, "key1", json_data["key1"])
flowFile2 = session.putAttribute(flowFile2, "list_value", i)
session.transfer(flowFile2, REL_SUCCESS)
session.remove(flowFile) Now I keep getting an error that I need to close the process session first because my original flowfile is still in use. How can I iterate over the list to create multiple flowfiles properly? I couldn't find any similar case or example of this sort. Thanks for tips and help. Edit: So I changed my code to create a dictionnary in order to use a new variable name for each flowfile created, but now I keep getting a key error when I want to transfer them to the success relationship. import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
from org.apache.nifi.processor.io import InputStreamCallback
flowFile = session.get()
if (flowFile != None):
stream_content = session.read(flowFile)
text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8)
json_data = json.loads(text_content)
d = {}
for x in range(len(json_data["list"])):
d["flowfile{0}".format(x)] = json_data['list'][x]
for file in d:
file = session.create()
file = session.putAttribute(file, "key3", str(json_data["key3"]))
file = session.putAttribute(file, "key2", json_data["key2"])
file = session.putAttribute(file, "list_value", d[file])
session.transfer(file, REL_SUCCESS)
session.remove(flowFile)
... View more
Labels:
- Labels:
-
Apache NiFi
04-07-2023
01:25 AM
Hello, is there such a thing as an escape character? I would need keynames with a "." in it. In this example something like: Output { "id": "3240", "files" : { "mac.File": "mac-20200806.json", "window.File": "window-20200806.json", "linux.File": "linux-20200806.json" } } But dots are interpreted as nesting objects. Thanks Edit: Nevermind, I found we can use escape characters with "\\"
... View more
04-07-2023
12:32 AM
2 Kudos
Just for everyones information; There was nothing wrong with the code. It was simply that the docker image I used to run Nifi did not build properly and therefore my modifications weren't taken into account.
... View more
04-04-2023
02:35 AM
Hello friends, I wrote a custom processor and now wanted to add a custom controller. The problem is, that the properties I wrote for the controller do not appear when I create the service. I followed the structure of this controller: https://github.com/Automna/NiFi/blob/master/nifi-rel-nifi-1.1.2/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java Here is what I did so far: public class MyNewControllerService extends AbstractControllerService implements MyControllerService {
public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
.name("Username")
.expressionLanguageSupported(ExpressionLanguageScope.NONE)
.displayName("Set username")
.required(true)
.defaultValue("User")
.description("Username to login to graph service")
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder().
name("Password")
.expressionLanguageSupported(ExpressionLanguageScope.NONE)
.displayName("Set password")
.required(true)
.defaultValue("password")
.description("Password to login to graph service")
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
public static final PropertyDescriptor URL = new PropertyDescriptor.Builder().
name("ApiUrl")
.expressionLanguageSupported(ExpressionLanguageScope.NONE)
.displayName("Set url")
.required(true)
.defaultValue("url")
.description("Full url to api")
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
private static final List<PropertyDescriptor> properties;
static {
final List<PropertyDescriptor> props = new ArrayList<>();
props.add(USERNAME);
props.add(PASSWORD);
props.add(URL);
properties = Collections.unmodifiableList(props);
}
OkHttpClient client = new OkHttpClient();
private static ObjectMapper mapper = new ObjectMapper();
@Override
protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
return properties;
}
@OnEnabled
public void onConfigured(final ConfigurationContext context) throws InitializationException {
final String user = context.getProperty(USERNAME).getValue();
final String password = context.getProperty(PASSWORD).getValue();
final String url = context.getProperty(URL).getValue(); What is missing that I only get a blank properties page?
... View more
Labels:
- Labels:
-
Apache NiFi
01-11-2023
07:58 AM
Hello, I cannot find an answer on the www to my following question. I have tried splitjson and jolttransform to achieve my desired outcome, but nothing does the trick. My input file looks like this: { "BVjsoBHDSI" : { "date" : "2021-04-08", "description" : "blabla1", "number" : 40 }, "FudNSk" : { "date" : "2021-09-26", "description" : "blabla2", "number" : 80 }, "Cfjdsl54fd" : { "date" : "2022-05-10", "description" : "blablubb", "number" : 31 } } Actually I just want to split the stupid thing, but since the root level has a different individual value nifi doesn't understand what I want to do. The idea was to use a jolt transform to add a key "id" with the original key as value. I can't figure out a proper specification though. Glad to hear your ideas. Desired output: { "BVjsoBHDSI" : { "date" : "2021-04-08", "description" : "blabla1", "number" : 40, "id" : "BVjsoBHDSI" }, "FudNSk" : { "date" : "2021-09-26", "description" : "blabla2", "number" : 80, "id" : "FudNSk" }, "Cfjdsl54fd" : { "date" : "2022-05-10", "description" : "blablubb", "number" : 31, "id" : "Cfjdsl54fd" } } Then I could easily overwrite the root and split the flowfile as usual.
... View more
Labels:
- Labels:
-
Apache NiFi
12-01-2022
02:33 AM
Hi, thanks for the details. Unfortunately it is not working. I get an empty array [] as output. I have tried it with extract and split mode. I applied the schema text property as suggested with "NestedKey" and "nestedValue" as name. None gives me an output. Meanwhile I have achieved what I wanted using SplitContent and then again another jolt processor. Of course it would be more elegant if I could make it work with ForkRecord.
... View more