Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 300 | 05-06-2026 09:16 AM | |
| 485 | 05-04-2026 05:20 AM | |
| 354 | 05-01-2026 10:15 AM | |
| 522 | 03-23-2026 05:44 AM | |
| 392 | 02-18-2026 09:59 AM |
09-30-2022
02:13 PM
@leandrolinof ListSFTP and FetchSFTP are commonly used to fetch files from a remote server. In order to use ListFile or FetchFile the location remote directory would need to be mounted as a local file directory. Matt
... View more
09-30-2022
02:09 PM
@PepeClaro Can you share your processor configuration and NiFi version being used? Thanks, Matt
... View more
09-30-2022
02:04 PM
@ImranAhmed If I am understanding you correctly, you are trying to verify the content has correctly been modified before it is written out via your invokeHTTP processor.Inspecting that your dataflow is working as expected. If you stop your InvokeHTTP processor after the ReplaceText processor, the FlowFile processed by the ReplaceText will queue in the connection. You can then right click on the connection and list the queue. From there you can view/download the content of that FlowFile post ReplaceText to verify it contains the expected modified content before the InvokeHTTP is started and writes it to your DB. Thanks, Matt
... View more
09-26-2022
03:36 PM
@ImranAhmed I am not clear ion what you are trying to do here. You are looking for some exact string and replacing with nothing? The search value is a java regex so what little i can see is looking for an exact string match which is being compared against the "entire text". So entire text means entire contents of your FlowFile is being loaded into NiFi's JVM heap also. Why "entire text" instead of "line-by-line"?
... View more
09-26-2022
03:29 PM
1 Kudo
@knighttime You should not be configuring any of your NiFi processors to us the Event Driven scheduling strategy. It was not moved from an experimental method to production ready. Advances in the Time rDriven scheduling strategy has made int more efficient. So Event Driven is pretty much deprecated at this point in time. If you are not using Event Driven scheduling on any processor component in your NiFi, You should not be setting a large "Maximum Event Drive Thread Count" pool (default is 5, but i recommend setting to 1). While you can increase the Maximum while NiFi is running, reducing will require you to restart your NiFi. Now when it comes to the "Maximum Timer Driven thread count" pool. We can create a large pool which is per node in your 4 node cluster (80 thread X 4). Then you configure concurrent tasks on your individual processors to scale concurrency on each processor component. Also keep in mind that many processors execute to check connection inbound connection queues and those thread may only be active for micro seconds before being released back to the thread pool. so actually seeing full thread utilization represented in the status bar of your NiFi's may be difficult to see. Tips about concurrent task setting on processors. Setting high concurrent tasks configuration across many processors can be worse than leaving everything set at 1 in terms of overall performance. Start in the basement (1 concurrent task) and slowly increment concurrent tasks on processors as needed. You mention data queueing up, but it is difficult to tell you why or provide guidance without seeing your dataflow and knowing which processors have data backed up to them in their inbound connections and the configuration of those processors. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-22-2022
11:04 AM
@wasabipeas I can think of no way to force delete if it is blocking on a revision mismatch between nodes. Nothing here has anything to do with version control. Is it always the same node reported in the pop-up message that fails to process the request? If so, have you verified the libs and version running on that one node match rest of cluster? If you go to the cluster UI and select "VERSIONS" tab, they all reflect same version? You could manually disconnect the one node that it keeps complaining about from the "NODES" tab. After it is disconnected, you could delete it from the cluster (Deleting the nodes does nothing flows or data on that node. It will require a restart of that one node to get it to rejoin cluster). Once the node is removed form your cluster (temporarily), your cluster should reflect 10/10 connected nodes now in the status bar of the canvas UI. Check to see if your are still having revision issues with the process after reloading the page. If all looks good, you could access the filesystem of the currently disconnected and deleted node, stop the NiFi service on that node, and delete/rename the flow.xml.gz and flow.json.gz files. Then start this node again. On startup, NiFi will inherit the flow from the cluster and in doing so get the cluster flows current revision for the problematic process group. If problem persists, restart node that was deleted so that it rejoins the cluster. Then disconnected the currently elected cluster coordinator. A new cluster coordinator will then be elected by zookeeper. Check to see if issue with process group is resolved. Reload your browser to force a page refresh. If issue is resolved, rejoin node to cluster via the cluster UI to see if issue returns. If so, we at least know which is our problematic node. You can of course, disconnect, delete, rename flow.xml.gz and flow.json.gz, and then restart node, just as we performed before so that flow is pulled from cluster on startup. If issue still persists, there is something unique about this node. Disk space ok?, any exceptions in logs?, while node may report same NiFi version, something different with contents of lib(s) folders (get a checksum and compare against other nodes). Hope this helps without needing to restart entire cluster, Matt
... View more
09-22-2022
06:33 AM
@Chhavi Your question is not very clear. is there any way I can integrate clustered nifi for storing and fetching the value Storing and fetching what value? NiFi includes a couple Redis controller services: RedisConnectionPoolService RedisDistributedMapCacheClientService For example, The RedisDistributedMapCacheClientService could be used by the PutDistributedMapCache NiFi processor to write values to Redis. The FetchDistributedMapCache NiFi processor could be used to fetch values from Redis. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-22-2022
06:24 AM
@ImranAhmed Can you share a screenshot of your dataflow and the configuration of your replaceText processor? You mention xml. Is your source File that you are trying to perform replace text on an XML format file? If so, that is not a text based (ASCII) content file. In that case, your configured search value is probably not matching on anything and this nothing gets changed in the original content binary content. NiFi is a content agnostic application. This means that NiFi can ingest any type of data. It does this by wrapping that content in to what NiFi calls a FlowFile. A FlowFile consists of two parts: 1. FlowFile metadata/attributes - Stored in the NiFi flowfile_repository. It contains details about the content such as filename, size of content, location of stored content, and any other attributes added by NiFi components (processors, controller services, etc) as the FlowFile traverse these components in your dataflow. 2. FlowFile content - Stored in claim files with in the NiFi content_repository. NiFi simple writes the binary contents to a claim and records the starting byte location and number of bytes of the content. This way NiFi does not need to be able to read the content to move it through a dataflow. It becomes an individual component's code responsibility for knowing how to read the content of a FlowFile. So NiFi includes processor components for many different data types. As far as XML content files, NiFi has limited native options (SplitXML, TransformXML, ValidateXML, XMLReader, and XMLRecordSetWriter). The latter two are controller services that could be used by processors like convertRecord. There is also the possibility that one fo NiFi's scripting processor could be used where the user writes a script that can read an handle the specific content type. There are execute processors that can execute and external command on the server where NiFi is running against the content of a FlowFile. So if there is an external command service that can take content input and return modified content back. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-22-2022
05:32 AM
@wasabipeas What version of NiFi-Registry is being used as well? In your NiFi UI, search for component UUID (a8db3982-1350-1b8b-ffff-fffff988699d). What kind/type of component is it? What is current state of the component (enabled, disabled, running, stopped, enabling, disabling, starting, stopping) Share screenshot of its current configuration. Thanks, Matt
... View more
09-22-2022
05:22 AM
@myzard Did your LDAP manager password contain any XML special characters? Did you verify ldapsearch worked from same same host were NiFi is installed using that manager DN and Manager password to get a return for the user you are trying to login in with? what output did you get from ldap search? For the ldap-provider, there are only two username and passwords in use: 1. Manager DN and Manager password configured in the ldap-provider 2. username and password entered at login interface. Other suggestions: - Make sure there are no leading or trailing whitespaces on the username or password configured in the provider or entered at the login windows. - Make sure the nifi.properties file is properly configured for the ldap-provider and not a different login provider like kerberos-provider. - Share you ldap-providers.xml file Thanks, Matt
... View more