Member since
07-29-2020
104
Posts
19
Kudos Received
16
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
65 | 06-28-2022 11:49 AM | |
38 | 06-28-2022 06:28 AM | |
164 | 06-16-2022 06:29 AM | |
53 | 06-15-2022 05:59 AM | |
78 | 06-14-2022 10:06 AM |
06-28-2022
11:49 AM
1 Kudo
Hi, It all depends how the output is getting generated. you probably need the processor that fetch the DB record to run on primary node first and then do load balancing downstream to process each record on a different node.
... View more
06-28-2022
06:28 AM
1 Kudo
Hi, Have you tried GenerateTableFetch processor. It can do incremental by saving maximum value for given column\s. Regarding the join why dont you create a view that does all the needed join and use that as an input for the GenerateTableFetch processor Table Name property.
... View more
06-20-2022
08:23 AM
Please check this old post on how to connect to mysql I assume its similar to oracle: https://community.cloudera.com/t5/Support-Questions/how-to-configure-and-connect-mysql-with-nifi-and-perform/td-p/109478
... View more
06-20-2022
06:40 AM
I apologize for that, I hope someone else step in and help you with better solution. May I just know why it did not help?
... View more
06-20-2022
06:25 AM
I thought we discussed this here: https://community.cloudera.com/t5/Support-Questions/How-to-load-json-record-to-postgres-as-json-datatype/m-p/345779#M234644
... View more
06-20-2022
06:11 AM
1 Kudo
This is not efficient and will use a lot of heap depending on how big the list and the json array. Can you explain more what are you trying to do and why you want to create different variable for each json flowfile? Maybe there is a better way.
... View more
06-20-2022
06:06 AM
1 Kudo
Hi, The string you are getting is not a valid json, so you need to use ExecuteScript Processor to parse the string and convert into array and then convert the array into json array. You can convert this string into array by removing curly brackets and then do split(',') then you can convert the array into json using something like: https://www.javaguides.net/2019/07/convert-set-to-json-array-using-jackson.html Pass the converted Json Array as new flowfile from the ExecuteScript Processor. Lets assume that your json array looks like this: ["1","1","1","3465689","B4:E6:2D:34:E1:D9"...] Then you use Json Jolt Processor and pass the following Jolt Spec: [ { "operation": "shift", "spec": { "*": { "@0": "Value-&0" } } } ] which will give the new flowfile as follows: { "Value-0" : "1", "Value-1" : "1", "Value-2" : "1", "Value-3" : "3465689", "Value-4" : "B4:E6:2D:34:E1:D9" ... } Hope that helps. If it does please accept solution. Thanks Samer
... View more
06-18-2022
08:54 AM
Hi, Which system did you install Nifi on ? If windows try to click any Java job running in the background and restart nifi.
... View more
06-16-2022
02:41 PM
By the way what is inside the users.xml file?
... View more
06-16-2022
02:01 PM
Sure, I hope they can help. By the have you seen my comments above about defining the nodes in the auotherizers.xml with the following format (case sensitive): CN=masternode, OU=NIFI CN=workernode02, OU=NIFI CN=workernode03, OU=NIFI
... View more
06-16-2022
01:26 PM
Just to be on the safe side, can you define the nodes in the authorizers.xml as follows. Keep in mind when you define something in the nifi config files its is case sensitive. CN=masternode, OU=NIFI CN=workernode02, OU=NIFI CN=workernode03, OU=NIFI
... View more
06-16-2022
01:21 PM
@MattWho, @araujo , can you guys help?
... View more
06-16-2022
12:54 PM
Its hard to tell what is causing the error , the only thing for me is just verifying that you followed the steps accordingly to setup the cluster. One thing you did not mentioned is creating the myid file under the folder ./state/zookeeper on each node that defines the node number as mentioned in the url I provided before, did you do that?
... View more
06-16-2022
09:14 AM
What the screenshot is telling that its still initializing the data flow, so give it some time and refresh.
... View more
06-16-2022
07:24 AM
Sorry I forgot to mention that in the PutSql processor set the Batch Size property to 1
... View more
06-16-2022
07:07 AM
Have you added the nodes in the authorizers.xml under userGroupProvider and AccessPolicyProvider
... View more
06-16-2022
07:04 AM
Im not familiar with the postgress sql but should not be like this: INSERT INTO table(column1, column2, …)
VALUES
(value1, value2, …); Also for the value you need to provide the attribute name , so it should be something like this: INSERT INTO table(jsonDataColumn) VALUES(${data});
... View more
06-16-2022
06:58 AM
Not sure if this is related , but if you are setting up a secure cluster the property nifi.cluster.protocol.is.secure should be true. If not just make sure you follow steps mentioned in the following url and you have not missed anything: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit
... View more
06-16-2022
06:29 AM
The way PutDatabaseRecord works is basically once it reads the json , it will try to match the json keys to the DB columns hence the error you are getting if not being able to map fields to columns, in your case it will be looking for columns: name & surename to store their values, so it wont store the whole json in one field. I had similar situation where I needed to store json and the only way I can think of is first load the json into an attribute using the ExtractText Processor, then use PutSQL processor and write the insert statement to the target table where the column is the Data and the value is the attribute contain the json value for example ${JsonDataAttribute}. You have to be careful how big your json is, if its too big then you have to figure out another way, but if its small then no harm from saving into an attribute.
... View more
06-16-2022
05:55 AM
Hi, Can you explain what happen when you use the PutDatabaseRecrod Processor?
... View more
06-15-2022
05:59 AM
Hi, It could be an encoding issue that I faced a while back. please check out this post: https://community.cloudera.com/t5/Support-Questions/Illegal-Arg-Exception-when-parsing-CSV-file-though-the-split/td-p/300565 Try to change the character set to UTF-16 to see if that will work. Hope that helps. Thanks
... View more
06-14-2022
10:06 AM
first you need to check if there is an attribute called filename on the original flowfile and its the same passed the output stream flowfile. If there is, then you should specify it as "filename" in the merge content property and NOT as "${filename}".
... View more
06-14-2022
07:42 AM
how is the original flowfile is generated. usually you should have filename attribute set by default on the original flowfile. this should be unique for the flowfile and this filename could be passed to the output stream, can you check that?
... View more
06-14-2022
06:59 AM
Is there a unique attribute on the original flowfile (like filename) that gets set on the output stream ? if so, you can set the mergeContent processor property " Correlation Attribute Name " to the attribute containing the unique value that is shared between original and output, this way you guarantee that only related original and output are getting merged.
... View more
06-14-2022
06:01 AM
Hi, Not sure I fully understand your question. Can you please provide screenshot of the flow you have and explain the expected output of each processor? From what I can understand you want to keep the original file but somehow you want to attach the output of the command stream to it. There is property on the ExecuteCommandStream Processor called " Output Destination Attribute " where you can store the output stream as an attribute on the original flow file. if you want to modify the content of the flow file after that based on the new attribute then you can use ExecuteScript Processor.
... View more
06-12-2022
06:03 AM
Hi, Can you give an example? I think if the url in the flowfile content you can use ExtractText processor where you can create dynamic property and use regex to extract the pattern you need. If the url in an attribute then you can use UpdateAttribute where you can also create dynamic property and use Expression Language function like find(regex) to extract the domain for a given attribute. Example ${FullUrl.find(regex)}. To set the regex for finding domain from url, checkout this link: https://regex101.com/r/jN6kU2/1
... View more
06-10-2022
02:46 PM
1 Kudo
Hi, regarding the first point I dont think you can do that through regex, this is more of transformation operation and you might need to use JsonJolt Processor to do transromation first to simplify the json and store all id's in an array then use regular expression to extract the ids. It will complicate matter it might affect performance. What kind of SQL are you using ? Do you have any json functions in that sql? If that is the case I would recommend you deffer this process to SQL and utilize sql function to parse json. For example in MS SQL you can use OPENJSON function to transpose json into table and then you can use this table in your query. For the second point, this is because the way the extractText reg expression work it treats like group, so beside the main attribute you will get an indexed attribute for each matched group. You can read more about this here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ExtractText/index.html You can also use this post to see if you can extract one attirbute. I just found about it: https://community.cloudera.com/t5/Support-Questions/Extract-whole-json-flowfilecontent-small-to-single-attribute/m-p/308960
... View more
06-10-2022
12:35 PM
Hi, If I understood your question correctly, you want to place the file content into an attribute and store it in sql? If that is the case you can use ExtractText Processor. there you add dynamic property lets call it "IdArray" and set the regular expression to "(?s)(^.*$)" to capture everything in the file. Just be careful there is a size limit set of 1 MB to be captured if you think your data will be more then you can increase it. Hope that helps, if so please accept solution. Thanks
... View more
06-10-2022
05:00 AM
I'm not seeing the load balancing being set on the success relationship of the DuplicateFlowFile. That will basically mean that the PutFile processor will be executed on each node (as many as the flowfile coming out of the duplicate processor.
... View more
06-10-2022
04:04 AM
Have you set the load balancing strategy on the queue upstream the put file to round robin. If that doesnt work for some reason, then your only option is to create a shared folder on all nodes, then you will have a workflow that picks up the file then you add as many PUTFile processors as the number of your nodes where each put file will save the file to a given node shared folder.
... View more