Member since
07-30-2019
3406
Posts
1622
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 210 | 12-17-2025 05:55 AM | |
| 271 | 12-15-2025 01:29 PM | |
| 248 | 12-15-2025 06:50 AM | |
| 293 | 12-05-2025 08:25 AM | |
| 488 | 12-03-2025 10:21 AM |
07-28-2021
10:57 AM
@Aiiyer What ERROR? All I see are a bunch of normal INFO log lines form the nifi-app.log snippet you shared. Please share the ERROR log line and stack trace if present. Or share screenshots of what you are doing and observing. Thank you, Matt
... View more
07-28-2021
10:53 AM
1 Kudo
@midee @janis-ax is correct that this is not possible. "ONLY" sensitive attributes can be encrypted/masked. Only component properties coded to as a sensitive property field would be able to decrypt an encrypted/masked value. If setting policies to block specific users from being able to view the data on these components, you may want to look at other options for performing these endpoint authentication/authorization actions. For example, with the NiFi endpoints you could use mutual TLS via certificates instead of token based authentication for accessing the endpoints. This does not mean you need to stop using or un-configure the token based authentication methods for access to your secured NiFi. If the endpoint you are trying to reach is not NiFi, you may want to see what other options it may offer for authentication that are not token based. Hope this helped, Matt
... View more
07-28-2021
10:40 AM
@midee You are using an InvokeHTTP processor to hit the rest-api endpoint of some authentication service to obtain a JWT Token, correct? The invokeHTTP processor gives you only two options for handling the response from the endpoint. Placed it in the content of the FlowFile or in to an attribute of the FlowFile. This means that any user who has been granted authorization to view the data of any component that this FlowFile will traverse can see the attributes and content of this FlowFile. There is no capability for the invokeHTTP to encrypt/mask the endpoint response in either options. Even if there was, this now masked/encrypted data would no longer be usable by downstream components. I suspect your plan is to then use this token to make additional requests to other endpoints later in your flow? Also keep in mind that a token generally has a limited lifespan. You should also have the ability to invalidate the token via another endpoint when you are done with it. So your options here are: 1. Limit user's access so they can not view the data in the dataflows that utilize this token so that they can't see the token in the content or attributes. If you are building the flow, then you know how to get the token so does not matter that you can see the token via the attributes or content. 2. Invalidate the token once done with it. Token can't be used anymore even if a user gained access to it later via looking at the content or attributes of a FlowFile. So you may consider building yoru flow to get a token --> use that token for other actions --> invalidate token (Using NiFi login authentication as a JWT token example, you hit the logout endpoint to invalidate the token on the server side. This means the client token will no longer work even if you have it still). Of course this means you flow needs to obtain a new token every time it runs. Issue here is if your flow utilizes same token for multiple FlowFiles. First FlowFile that hist logout will invalidate token other FlowFiles may try to still use. Note that the default life of a token issued by NiFi is 12 hours. After that the client token can't be used anymore and a new token would need to be obtained anyway (of course this age can be adjusted via the NiFi login provider configuration) If you found this answered your query, please take a moment to login and click "Accept" on the solutions that assisted you. Thank you, Matt
... View more
07-28-2021
10:18 AM
1 Kudo
@pritster5 The screenshot shared does tell us some the following: We can see on the AzureBlobStorage "0(2)" displayed in the upper right corner. This tells us that this processor still has two active threads that have been "terminated" manually by a user (Meaning user tried to stop processor but it failed to stop so the user selected terminate). Terminating threads simply tells NiFi to release the FlowFile associated to that thread back to the inbound connection queue it belongs to and set processor to send anything returned by those still running threads that are marked as terminated to null. So since we can tell you still have running threads, the question becomes... "What are these threads doing?" "Are these thread hung?" "Are thee threads just very long running?" I would suggest using executing following several times (1 - 3 minutes apart) to get a series o thread dumps: ./nifi.sh dump <filename fo dump> Then you should inspect those threads to see perhaps what they are waiting on or if they are making any progress (stack changes) between each thread dump. This will answer your questions above. Maybe connection to AzureBlob Storage is very slow? Maybe Host resources are high and your threads are waiting on CPU? etc... If you found this assisted with your query, please take a moment to login and click "Accept" on solutions that helped you. Thank you, Matt
... View more
07-28-2021
10:07 AM
2 Kudos
@robnew In addition to what @janis-ax suggested about setting bulletin level to none on the processor to stop WARN and ERROR level log output from that processor from being sent to the bulletin board, you could also suppress the logging of this processor class via the NiFi logback.xml file. Changing the bulletin level on a processor does not disable logging being produced by your processor code. That logging will still find it way in to the NiFi logs via the logback.xml even with bulletin level set to known. So though processor you can stop the bulletins and through NiFi's logback.xml you have additional options to send logging produced by your processor class to its own appender or one of the existing appenders already defined in that file. If you found the feedback provided to your query helped, please take a moment to login and click "Accept" on all solutions that helped. Thank you, Matt
... View more
07-13-2021
12:58 PM
@midee The invokeHTTP processor will not execute against multiple FlowFiles collectively. You need to have one FlowFile that contains both your token in a FlowFile attribute and the desired content you are trying to post in the content. The dataflow that obtains the token could be merged with the other dataflow that has yoru content. If you have multiple FlowFiles that will need to use the same token, yo could have to separate dataflows. One that gets the token and rights it to a distributed Map Cache via the PutDistributeMapCache processor. Second dataflwo the gets your content file and fetches the token via the FetchDistributedMapCache processor. Then you have one FlowFile with the token you need in an attribute and the content on the FlowFile to be transferred via InvokeHTTP. If you found this assisted with yoru query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
07-13-2021
12:48 PM
@chams221 NiFi's UI will not be available immediately after you start the service. The UI will not be available until the child process has finished loading. You can monitor teh progress by tailing the nfii-app.log. Once you see the log line that states: 2021-07-13 19:46:43,855 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2021-07-13 19:46:43,855 INFO [main] org.apache.nifi.web.server.JettyServer http://127.0.0.1:9090/nifi
2021-07-13 19:46:43,858 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2021-07-13 19:46:43,858 INFO [main] org.apache.nifi.NiFi Controller initialization took 44494103250 nanoseconds (44 seconds). then you should be able to access the NiFi UI provided you can reach that address from your client browser. Thank you, Matt
... View more
07-13-2021
12:38 PM
@Ash1 Not clear from yoru query how files are getting into or out of your NiFi. Assuming you have already received a set fo FlowFiles for the day in to your NiFi dataflow, the best approach may be to notify if any fails to be written out /transferred at the end of your dataflow. In this manor not only would you know that not all Files transferred, but would know exact what file failed to transfer. There are numerous processors that handle writing out FlowFile content (transfer) to another source or local file system. Those processing components typically have relationships for handling various types of failures. These relationships could be sent through a retry loop via the RetryFlowFile [1] processor back to the same transfer processor that failed. You define in the RetryFlowFile processor how many times you want a FlowFile to traverse this loop. After X number of loop it would get routed out of the loop to your PutEmail [2] processor where you could dynamically set the email content to include attributes from that FlowFile like filename, hostname of NiFi that failed to transfer it, etc... From the PutEmail processor you could send that FlowFile to somewhere else for holding until manual intervention was taken in response to that email. [`1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RetryFlowFile/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.PutEmail/index.html If you found any of the response given here assisted with your query, please take a moment to login and click "Accept" on each of those solutions. Thank you, Matt
... View more
07-13-2021
06:46 AM
@Indhu The more detail you provide the better community members can assist with your query. Can you share an example input and the desired output date string? What format is your source date in? How are you ingesting that data in to NiFi? Thank you, Matt
... View more
07-12-2021
01:27 PM
@LucasMorgado I don't see any ERROR? Was your old NiFi secured? Is the new NiFi secured? Along with your flow.xml.gz, did you also copy over your users.xml and authorizations.xml files to your new installation? Your screenshot indicates that your user is not authorized to see any of the components (Processors, Process groups, labels, etc), so they are all ghost implementations. Make sure your authorizers.xml is setup just like your old NiFi and that the above mentioned files were also copied over to your new install. NIFi only reads these files on startup. If you found this addressed yoru question, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more