Member since
02-01-2022
274
Posts
97
Kudos Received
60
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
222 | 05-15-2025 05:45 AM | |
2999 | 06-12-2024 06:43 AM | |
5113 | 04-12-2024 06:05 AM | |
3418 | 12-07-2023 04:50 AM | |
1894 | 12-05-2023 06:22 AM |
04-26-2022
06:24 AM
2 Kudos
@jonay__reyes I think by default you will see the result you are expecting, however, the expected limit of 5 concurrent connections may be a challenge. Let's address your questions first: Does this translate to simply using 1 InvokeHTTP processor configured to 5 "Concurrent Tasks" and that's it? - 1 proc w/ 5 concurrent tasks, will provide what is in effect 5 instance copies and they can run more than 5 requests each if there are ample flowfiles queued up. So, NO. For your use case, i would recommend that you set it to 1, and control the # of flowfiles upstream. Will the processor wait for the remote endpoint's request before sending the next one? YES if concurrent task set to 1. NO, if set higher (2+) they will execute in parallel How does the "Run Schedule" works together with the previous settings? (if I had, e.g.: 1 sec). Run Schedule sets how long a process will operate before a new instance is necessary. If the request/response times are low, this setting will allow you to push more data through each instance without creating separate processes for each. If the request/response time is high, you can use this to help with long execution. Experiment carefully here. I've been proposed with splitting the incoming queue and put 5 InvokeHTTP processors in parallel, each one attending 1/5 of the incoming flowfiles (I'd do the pre-partitioning before with some RouteOnAttribute trick), but I think it's exactly the same outcome as the 1. above. Is it? Correct, there is no reason to do this, avoid duplicating processors For concurrent tasks and run schedule adjustments, you should always experiment in small increments, changing one setting at a time, evaluating, and repeating until you find the right balance. I suspect that you will not need 5 long executing request/responses in parallel, and that even with default settings, your queued flowfiles will execute fast enough to appear "simultaneous".
... View more
04-26-2022
06:06 AM
@Tokolosk I would recommend that you provide java's cacerts as the keystore/truststore with the default password (changeit). This will work for most public certs. If you have a custom cert at the SFTP end you will need to import that cert and create your own keystore/truststore. I always try cacerts first.
... View more
04-25-2022
01:31 PM
@Tra The recommend path forward here is to use a JSON Record Reader and JSON Record Writer with schema matching your source (reader) and downstream needs. You want to avoid splitjson and regex matching flow file contents.
... View more
04-08-2022
07:10 AM
@mala_etl Well I cant do all of them, that would be to your values, not mine, but to achieve the move, you need to set: Completion Strategy: Move File Move Destination Directory: The directory to move file to Create Directory: enable true/ disable false Be sure to check the ? for each property, it will explain everything.
... View more
04-07-2022
05:15 AM
2 Kudos
@krishna123 this is how nifi is supposed to work. If there is no input, even though the processor is Play/Green, it is not doing anything. It is only "running" if there is input.
... View more
04-06-2022
06:04 AM
1 Kudo
@mala_etl Check out this feature in FetchSFTP: It allows you to move the files into a new directory after fetching them.
... View more
04-06-2022
05:54 AM
@krishna123 NiFi data flows are expected to be operated in an always on capacity. If no input is arriving, the processor is not doing anything, there is no reason to stop it. Can you explain in more detail why you want it actually stopped?
... View more
04-06-2022
05:37 AM
Ahh that certainly is a different challenge which would require a slightly different approach. My best recommendation, outside of making your own processors, would be a combination of api calls to prepare variables for a processor group that uses listsftp and fetchsftp or getsftp once the variables are setup.
... View more
04-05-2022
09:15 AM
1 Kudo
@mbraunerde Before GetSFTP, you need this processor first: ListSFTP ListSFTP will send GetSFTP the the list of files to get. Be sure to search in your NiFi Processor window, SFTP, to see all matching processors. This works for any keyword, and I use it often. Docs here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.16.0/org.apache.nifi.processors.standard.ListSFTP/index.html
... View more
03-23-2022
02:29 PM
@Nifi_Al Use getFile/putFile to read and write in your flow. Then If you use unpackContent against a zipped flowfile it should create individual artifacts flowfiles which can be additionally unzipped further. Alternatively, if you can create a script to unzip recursively, you can use nifi to executeScript. Maybe even use executeProcess too.
... View more