Member since
07-30-2019
3467
Posts
1641
Kudos Received
1016
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 119 | 05-04-2026 05:20 AM | |
| 430 | 03-23-2026 05:44 AM | |
| 327 | 02-18-2026 09:59 AM | |
| 575 | 01-27-2026 12:46 PM | |
| 1007 | 01-20-2026 05:42 AM |
02-14-2020
09:55 AM
1 Kudo
@vikrant_kumar24 The ExecuteStreamCommand processor is not doing anything fancy here. It is just trying to execute the command you enter. Assuming the command was "myscript.py", The executeStreamCommand would be doing same thing as you would if you opened a terminal window on your NiFi node and clicked on "myscript.py" to run it. Keep in mind that NiFi is performing this action as the user that owns the NiFi java process. The ExecuteStreamCommand processor will pass the FlowFiles content (if there is any) to stdin. Have you perhaps looked at the ExecuteScript processor instead? Hope this helps, Matt
... View more
02-14-2020
09:50 AM
1 Kudo
@DavidR NiFi requires that a sensitive properties key has been configured. The Sensitive properties key is used to encrypt all passwords written to the flow.xml.gz file. The flow.xml.gz is not portable between different installations of NiFi unless at least 1 of the 2 following is true: 1. Both NiFi installations are using the exact same sensitive properties key in the nif.properties file: nifi.sensitive.props.key= All the sensitive properties (passwords) entered in to components configured in your dataflows contained within the flow.xml.gz are encrypted. NiFi will fail to start if it cannot decrypt these sensitive properties as the flow.xml.gz is uncompressed and loaded in to memory. 2. The flow.xml.gz contains no encrypted sensitive properties. If you do not know the sensitive properties key used on NiFi where the flow.xml.gz was obtained, you can uncompress the flow.xml.gz yourself and remove all occurrences of "enc{.*}". For example: <value>enc{2fecf4bbb0456fd10c088f73aab2b3c3e92b532afa46eb042f0b0a14b06b1b60}</value> would be replaced with: <value></value> Then gzip the flow.xml again. This removes all passwords from your flow.xml.gz will allowing it to be loaded by any new NiFi installation. You will then need to go through your components and set the passwords again which will then get encrypted to the flow.xml.gz using the sensitive props key set on that NiFi. Hope this helps, Matt
... View more
02-13-2020
05:38 AM
@vikrant_kumar24 If you open a terminal window on your Windows server, can you execute your python script? If you are not successful from command line, the ExecuteStreamCommand processor is not going to be successful as well. Thanks, Matt
... View more
02-13-2020
05:31 AM
1 Kudo
@Umakanth NiFi components (Processors, controller services, reporting tasks, etc.) that have password properties do not support retrieving these passwords from an external source or service. NiFi not only obscures the passwords in the UI, but also encrypts all those passwords when they are written to disk in the flow.xml.gz. Additionally, for passwords entered in the various NiFi configuration files, NiFi offers and encrypt config toolkit that can encrypt all these sensitive properties in these configuration files on disk. I suggest maybe opening an Apache NiFi Jira with details around what you are trying to accomplish here for a possible future feature. - For passwords utilized through NiFi dataflow components, my thought here would be maybe around a NiFi controller service for connecting to such external services. This would also require that processors that would need to use this new NiFi Controller Service to retrieve passwords would all need to be modified as well with new configuration properties to interface with the new controller service. This is by no means a simple change in NiFi, but getting the idea out there with some strong use case for it can get the ball rolling in the community. Hope this helps, Matt
... View more
02-12-2020
01:15 PM
@chhaya_vishwaka It may be beneficially to other users since this thread/topic was so long, if you accept each response that helped directly with getting to your final working cluster. I am happy to hear that you are up and running. Now you get to explore the very granular authorizations/access policies NiFi provides within your secured cluster. Matt
... View more
02-12-2020
01:11 PM
@thuylevn The specific processors you mention were never part of any distribution of Apache NiFi. 1. JSONToAttributes: Typically when extracting json elements in to FlowFile Attrinutes you would use the EvaluateJsonPath processor. There is an "AttributesToJson" processor, is that maybe what you mean? 2. ConvertJSONToAvro NiFi ships with a ConvertAvroToJson processor, is that perhaps what you mean? If you trying to convert from one content format to another, the preferred processor to use would be "convertRecord". In this processor's configuration you would select a Record Reader which reads the current format of the inbound FlowFile's content (JsonPathReader or JsonTreeReader in this case) and you then select a Record Writer (AvroRecordSetWriter in this case) which writes the content for the outbound FlowFile. If the processor types you mentioned did exist for you, then they were added via a custom nar to your previous NiFi installations. Hope this helps, Matt
... View more
02-11-2020
10:34 AM
@JatinSab Troubleshooting this here with limited information is going to prove difficult. Does the downstream processor of the connection with the "stuck" FlowFiles have... 1. Configured for all nodes or primary node Execution? If primary node only, are the queued FlowFiles on the primary node? 2. Does processor show it has active threads? (small number in upper right corner of processor) . If so, get some thread dumps from your NiFi and inspect if the thread is progressing or hung. 3. Is back pressure being applied on any of the outbound connections? 4. Is processor getting threads to do work? What is CPU load on your system and configured "Max Timer Driven Thread Count?" It may be set to low causing a thread starved situation for your dataflow. Additionally, if you try to perform a "list queue" on the connection of issue, what result do you get? If you have a support contract with Cloudera, you can open a support case and they can help you to a solution. Thanks, Matt
... View more
02-11-2020
08:27 AM
@venu413 Are these the same FlowFiles that were stuck prior to the upgrade? If you clean out your connections, do you see the issue occur later?
... View more
02-11-2020
08:20 AM
@JatinSab Apache NiFi 1.11.1 specifically has a fix https://jira.apache.org/jira/browse/NIFI-7059 which introduced a bug with load-balanced connections. This bug is addressed in 1.11.2 and is covered in jira https://jira.apache.org/jira/browse/NIFI-7117. Thanks, Matt
... View more
02-11-2020
08:04 AM
1 Kudo
@chhaya_vishwaka Please stop your NiFi nodes and delete the local NiFi "state" directory on each of them. Then restart your NiFi nodes to see if this helps. Removing the local state will clear out the retained node information which may still contain info about your nodes before they were secured. Thanks, Matt
... View more