Member since
07-30-2019
3406
Posts
1622
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 196 | 12-17-2025 05:55 AM | |
| 257 | 12-15-2025 01:29 PM | |
| 191 | 12-15-2025 06:50 AM | |
| 285 | 12-05-2025 08:25 AM | |
| 476 | 12-03-2025 10:21 AM |
07-12-2021
01:19 PM
@midee So the output from first invokeHTTP is your token response. You could use the PutdistributedMapCache processor to write that token to a cache entry Then in your other flow with the GetFile processor, you send success to a FetchDistributedMapCache processor to retrieve that token and write it to an attribute (token for example) on this FlowFile. Now you have your token in an attribute on the same FlowFile that has your content for the next invokeHTTP processor in your flow If you found this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
06-29-2021
05:17 AM
@data-light NiFi nodes will always be authenticated via the clientAuth certificate they present. The Owner DN from the node's PrivateKeyEntry in the keystore file is what is used. What is important to know is that NiFi provides a mechanism that can trim/modify those full DNs through the uses of identity mapping properties [1]that users can configure in the nifi.properties file. After a mapping is applied against an identity string the resulting mapped value is what is actually passed to the configured NiFi Authorizer. So within the authorizer, that now mapped value is what the authorization policies need to be mapped to. The nifi-user.log will show the mapped value (case sensitive) that is being used during authorization. So if you see only "Nifi-node-0" in the nifi-user.log, then that exact string is what needs to exist in your authorizer and have policy assigned to it. If the nifi-user.log was showing "cn=Nifi-node-0, ou=mycompany, ou=nifi", then that full DN would need to be present in your authorizer with assigned policies. So you will want to get a verbose output of your nodes keystore files, your nifi.properties file to see what identity mapping patterns have been set, and the authorizers.xml to see how your authorization setup is done, and the nifi-user.log. [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties Hope this helps address your question. IF so, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
06-24-2021
05:34 AM
@midee I just ran the test myself with your sample as a one line formatted json and it still worked. {"expand": "renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id": "1155391","key": "DPGAUT-2","fields": {"customfield_10001": null,"customfield_10002": null,"customfield_10003": null,"customfield_10004": null,"customfield_10005": null,"customfield_10006": null,"resolution": null,"customfield_10101": "","customfield_10007": null,"customfield_10008": null,"customfield_10009": "This is required value","customfield_10010": null,"customfield_10011": null,"customfield_10012": null,"customfield_10013": null,}} Make sure your RouteOnContent processor dynamic property value does not have a line return in it. You'll notice it only shows "1" line above. If you see a "2" line then you have a line return in your Value field which would cause the issue you are seeing. Hope this helped, Matt
... View more
06-24-2021
05:22 AM
@Mahi123 Just to make sure i understand your use case correctly, you want to change the content only of your NiFi FlowFile? You have a NiFi FlowFile with following json content: {
"name": "Jon",
"title": "king of the nights watch"
} and you want to modify it to: {
"name": "Jon_<current date>",
"title": "king of the nights watch"
} I read your statement "I want to change the name of the file" as wanting to edit the filename and not the content of the file itself. Which is something totally different than above. Assuming you are looking to modify content and not the filename itself, you can do this using the ReplaceText processor after your EvaluateJsonPath processor. Search Value = ("name": "${name}") Replacement Value = "name": "${name}_${now():format('yyyy-MM-dd-HH-mm-ss')}" Replacement Strategy = Regex Replace Evaluation Mode = Line-by-Line NiFi typically has more than one way to satisfy the use case and there may be better option than the above using the various available record based processors. You may want to explore those as well. If you found this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
06-18-2021
09:15 AM
@techNerd Questions for you: 1. Is Computer B consuming all the files LocalFolder and from every sub-directory within c:\users\LocalFolder? If yes, then your SFTP processor should be configured to Search Recursively. 2. Once consumed by computer B, are the original file being left on Computer A? If yes, you do NOT want to use the GetSFTP processor as it will refetch the same files over and over again each time it executes since they are not being removed from Computer A. A typical dataflow here would look like this: The ListSFTP processor would be configured to use as follows: The "Tracking Entities" listing strategy will help when source files may have older timestamps than files previously listed from a different directory. Using this listing strategy requires you to set a "Entity Tracking State Cache". There are multiple cache services to choose from. I simply used the "DistirbutedMapCacheClientService" controller service which gets configured to point at a "DistributedMapCacheServer"controller service I also setup within the same NiFi. The "Remote Path" is set to the top most folder you want to start listing files from ( C:\User\LocalFolder\ ). The "Search Recursively" property is set to "True" so that any random new sub folders added are also searched. The "Success" relationship is then routed via a connection from the ListSFTP to the FetchSFTP processor. The FlowFiles produced by the ListSFTP processor will have numerous FlowFile Attributes set on them that will be used later to persist your directory structure and to fetch the content via the FetchSFTP processor. You can list the flowfiles on a queue and click the "view details" icon next to nay listed FlowFile to see the attributes currently assigned to that FlowFile. For example: filename = testfile-32.txt
path = /tmp/LocalFolder
sftp.remote.host = <SFTP server hostname>
sftp.remote.port = 22 Then if you look at attributes for file from an added SubFolder: filename = testfile-31.txt
path = /tmp/LocalFolder/SubFolder
sftp.remote.host = <SFTP server hostname>
sftp.remote.port = 22 The FetchSFTP processor then uses these attributes set on each FlowFile to fetch the actual content from the source SFTP server: Downstream in your dataflow after you have fetched the content, you can still use the Attributes on each FlowFile when working with yoru FlowFiles. For example using the "path" attribute to dynamically control where a FlowFile will be written on a target system (might be the local NiFi via PutFile processor, another SFTP server somewhere else, etc.). If you found this addressed your query, please take a moment to login and click "Accept" on all provided solutions that helped. Thank you, Matt
... View more
06-17-2021
07:02 AM
@bhavya The more detail you provide along with example of what you are trying to accomplish in yoru use case may help you get more involvement from the community on your query. "SQL file"? Are you trying to execute an SQL query and write the results returned by that query in JSON format? If so, you'll want to take a look at the ExecuteSQLRecord [1] processor and the JsonRecordSetWriter [2] controller service. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.ExecuteSQLRecord/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.13.2/org.apache.nifi.json.JsonRecordSetWriter/index.html Thanks, Matt
... View more
06-17-2021
06:50 AM
1 Kudo
@techNerd Based on yoru example, you can actually modify your XML using the ReplaceText [1] processor. The processor can be configured with Java regex that would match on the string you want to remove and the replacement value would be set to an empty string. Your regex pattern in this case would need to match on (don't forget the spaces): ( unit = ".*?") The parenthesis mark the java regex capture group. .*? is a non greedy match on any character for zero or more characters until the very next quote. This allows this to match on any variety of unit strings ( meter, kilogram, and lb) from your example This will result in the following: <particular>
<name>Nicholas</name>
<height unit = "meter">183</height>
<weight unit = "kilogram" unit = "lb">75</weight>
</particular>
<particular>
<name>Debbie</name>
<height unit = "meter">163</height>
<weight unit = "kilogram" unit = "lb">45</weight>
</particular> being converted to: <particular>
<name>Nicholas</name>
<height>183</height>
<weight>75</weight>
</particular>
<particular>
<name>Debbie</name>
<height>163</height>
<weight>45</weight>
</particular> [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.ReplaceText/index.html If you found this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
06-16-2021
06:26 AM
@techNerd I am looking at your example and am not clear on what your are "removing"? looks like you are adding text to your example. Thanks, Matt
... View more
06-16-2021
06:19 AM
@tech Please share a little bit more around your use case. The NiFi SFTP based processors are designed to create a FlowFile for each File retrieved from the target SFTP server and path. You want to retrieve a directory. NiFi has to write output from the retrieval to a FlowFile, so what are you looking to have written to that FlowFile? Then what do you want NiFi to do with folder once it is retrieved? More typically what is done in this situation is the ListSFTP [1]/FetchSFTP [2] (used when using NiFi cluster) or GetSFTP [3] (used when it is a standalone NiFi) processor are configured to retrieve all files from within a target folder on the SFTP server. One FlowFile is produced for each FlowFile that is retrieved. These processors all write FlowFile attributes to the produced FlowFiles. This includes the "path" attribute that tracks the directory/folder the files were fetched from. NiFi Expression Language (NEL) [4] is what is used to interact with these FlowFile attributes within your NiFi dataflow(s). Even if you did this outside of NiFi, you would get the folder and all files within it. Using above path attribute, each FlowFile knows folder name, so you could use that FlowFile attribute to write those FlowFile out to a target within the same original folder name. So same outcome can be accomplished. You can then do many things depending on your use case with these files once they are retrieved. Maybe use a mergeContent processor to create a tar file with all files added to it.... Maybe write these FlowFiles to a new target destination within a folder of same name as source... etc. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.ListSFTP/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.FetchSFTP/index.html [3] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.GetSFTP/index.html [4] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html If this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
06-16-2021
05:52 AM
@anoop_u_shet @Ash1 Let me try to answer both question here since they are related. 1. Can you set multiple java regular expressions in on single ListSFTP processor? No If you can come up with one single java regex, that can do your filtering, that will work. 2. Can you have one ListSFTP processor connect to two SFTP servers running on two different target servers? No. You would need to have one ListSFTP processor for each target SFTP server. 3. Can I have just one FetchSFTP processor with multiple ListSFTP processors feeding it? Yes *** The ListSFTP processor is designed to create FlowFile with numerous FlowFile attributes/metadata and no content. That metadata is then used by the FetchSFTP processor to retrieve tat FlowFile's content. *** If each target SFTP server uses passwords to authenticate, you may need to use different fetchSFTP processors since passwords can not be stored in FlowFile attributes. Take a look at the ListSFTP [1] processor documentation/usage (available within yoru install directly from UI under help or on Apache NiFi site) and look at the list of Attributes it writes to the NiFi FlowFiles it creates. You'll see below as some of those attributes: sftp.remote.host sftp.remote.port filename path In the FetchSFTP [2] processor, these Attributes from the FlowFile should be used to fetch that FlowFile's content using NiFi Expression Language (NEL) [3]. NEL is commonly used through out your dataflow(s). It allows utilize attributes from FlowFiles (among other things) in processors. For example, your FetchSFTP may look like this: You'll see 4 places where I used NEL. NEL expressions always start with "${" and end with "}". Above examples are very simple in that they only read the attribute name form the source FlowFile and use the value assigned to that FlowFile Attribute in the processor configuration property. NOTE: Not all processor properties support NEL. Floating your cursor over the "?" icon in a property will tell you via popup if NEL is supported by that property. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.ListSFTP/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.FetchSFTP/index.html [3] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html If my responses assisted with yoru queries, please take a moment to click "Accept" on all responses that contributed to a solution for you. Thank you, Matt
... View more