@ajaykumardev32  I would try to redesign your dataflow to avoid splitting the FlowFiles produced by tailFile processor.  NiFi FlowFile content is immutable (can not be modified once created).  Anytime the content of a FlowFile is modified, the new modified content is written to a new NiFi content claim.  If the processor has an "Original" relationship, an entirely new FlowFile is created (both metadata and content).  Those without "original" relationship that modify FlowFile content will simply update the existing FlowFile's metadata to point to new content claim.  So your SplitText processor is producing a lot of new FlowFiles. You then have inefficient thread usage downstream where processor are now executing against many small FlowFiles. As far as Provenance repository goes, you can configure the max amount of storage it can use before purging older provenance events.  Content and FlowFile repositories should not be on same disk since it is possible for content repository to fill the disk to 100%. You want to protect yoru FlowFile repository from filling to 100% by having it on a different physical or logical drive. Try utilizing the available "Record" based processors instead to avoid splitting FlowFiles and to do the record conversion/transform/modification.  In your case take a look at these record processors to see if they can be used in your use case: UpdateRecord 1.28.0 ConvertRecord 1.28.0 JoltTransformRecord 1.28.0 ScriptedTransformRecord 1.28.0 You are already using a "record" based processor to write to your destination DB. Other strategies involve adjusting the "Max Timer Driven Thread" and  processor "concurrent tasks" settings.    You'll need to carefully monitor cpu load average as you make incremental adjustments. If you max out your cpu, there is no gain from adjusting higher anymore.  Setting "Concurrent tasks" too high on any one processor can actually lead to worse performance overall in your dataflow.  So small increments and monitor is the proper path to optimization in this area. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Shrink  Not sure why you would want to "control processors" within dataflows in NiFi.  This is not typically a good design choice. From the image shared, I see the NiFi URL is "http" and not "https".  If your have you NiFi setup unsecured (HTTP), no NiFi authentication is going to be used even if setup in the NiFi core configuration files.    If your NiFi was secured (HTTPS), you would still need a StandardRestrictedSSLContextService to at least provide a truststore that contained the complete trust chain for the ServerAuth certificate used in the NiFi nifi.properties keystore to establish the 1-way TLS connection before you would be able to redirect to the oauth2 endpoint to get a token.  The other issue is you are not fetching a token from your oauth2 provider, but rather trying to fetch a token from whichever NiFi login provider you have configured.  That is because the "rest-api/access/token" endpoint is used for the NiFi login providers (ldap-provider or kerberos-provider). When I use InvokeHTTP processor to access NiFI's rest-api, i always use MutualTLS authentication.  I find it the easiest because you don't need to worry about managing tokens or fetching new tokens each time the expire.  To support mutualTLS authentication, a StandardRestrictedSSLContextService would need to be used in the InvokeHTTP processor and Your NiFi would need to be secured (HTTPS).  Your secured NiFi will have a keystore and truststore setup in nifi.properties file.   NiFi out-of-the-box will generate generic self-signed keystore and truststore files for you. I strongly encourage you to use properly signed certificates in production.   Simplest approach to set this up is to simply use the same keystore and truststore from the nifi.properties in the StandardRestrictedSSLContextService you'll use with the invokeHTTP processor.  You'll need to make sure that the NiFi ClientAuth certificate DN from the keytsore is properly authorized for the NiFi rest-api endpoints you want to use (this of course also means you are NOT using the out-of-the-box single-user-authorizer and instead using an authorizer that allows you manage user authorizations manually like the StandardManagedAuthorizer.) NOTE: the Apache NiFi 2.0.0 releases where tech preview.  NiFi 2.1+ are the official releases of the new 2.x line.  2.0.0-M4was last TP release, so it is going to be pretty close to the first GA release. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nifier  Not much insight I can offer here without you sharing the full nifi-api request being made. Are you using curl? How are you passing your user authentication in the rest-api call?  (The exception you shared points at an authentication issue) Thank you, Matt ... View more

@nifier  I am not seeing the same behavior as you have reported which make me think your extracted.path is not not really 0 bytes. I suggest adding another dynamic property to your update Attribute processor to output the calculated length to verify. Here is what i see after my UpdateAttribute processor (newdir is set using your shared NiFi Expression Language (NEL) statement). So I created  extracted.path with a single white space and then see what you describe, but that whitespace is a byte, so expected output: If it turns out your problem is because you have white space in the Extracted.path attribute, you could modify your NEL statement as follows: ${extracted.path:trim():length():gt(0):ifElse(${extracted.path:append('/'):append(${filename})},${filename})} You use the trim() NEL function to remove leading or trialing whitespace from the extracted.path before calculating length.  This includes trimming of a line return. So in below you will see desired output even though extracted.path length is not 0 because I trimmed white space or line return. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@sha257  You already shared your authorizations.xml file which shows where you are lacking the needed NiFi node certificate authorization for proxy user requests.  NiFi will attempt connect to NiFi-Registry using the NifiRegistryFlowRegistryClient setup in NiFi.   The NifiRegistryFlowRegistryClient has two configuration properties: The URL is set to "https://<NiFi-Registry Hostname>:<NiFi-Registry-port>/" The SSL Context Service if NOT set will use the keystore and truststore setup in the nifi.properties file.  The keystore and truststore used (whethere defined via an SSL Context Service or  using those from nifi.properties file MUST be capable of successfully negotiating a MutualTLS exchange between the NifiRegistryFlowRegistryClient and NiFi-Registry.   This means the ClientAuth Private key in the keystore must be trusted by the truststore configured in NiFi-Registry.properties file. This also means the ClientAuth PrivateKey entry in the NiFi-Registtry keystore file must be trusted by the truststore in the NifiRegistryFlowRegistryClient.  If this mutualTS exchange is not successful, the connection with your NiFi-Registry will be as the "anonymous" user identity. You could use the output from openssl to see if mutualTLS connection would be possible by looking at output from both below: openssl s_client -connect <nifi-registry-hostname>:<nifi-regsitry-port> -showcerts openssl s_client -connect <nifi-hostname>:<nifi-port> -showcerts Above assume your NifiRegistryFlowRegistryClient is using the same keystore and truststored used in the nifi.properties file. Alternatively,  you could get the verbose listing from your  NifiRegistryFlowRegistryClient configured keystore and trustore and compare those with the openssl output from NiFi-Registry to validate if a mutualTLS connection is possible.  ------------------------------ Assuming above is all good, then you need to verify proper authorizations are setup to allow the NifiRegistryFlowRegistryClient client identity (derived from the privateKeyEntry DN from the NifiRegistryFlowRegistryClient keystore) has the proper authorizations setup in NiFi-Registry. The NifiRegistryFlowRegistryClient keystore PrivateKey DN must eb authorized in NiFi-Registry for the following policies: "Can Manage buckets"  (Read)  "Can proxy user requests" (Read, Write, Delete) Above allows the NifiRegistryFlowRegistryClient to be able to see all the NiFi-Registry buckets and to make Read, Write, or Delete request on behalf of the user identity (user string shown in upper right corner of NiFi Ui) authenticated in NiFi. Once above is all good, we move on to what permissions the NiFi user that is being proxied needs in NiFi-Registry. The user identity need to be authorized on each bucket from which or two which flow definitions will be read or written. Here is a Sample public bucket ("Make publicly Visible" is checked): Since this bucket as "Make publicly Visible" checked, any user identity and anonymous users can read all flow definitions added to this bucket (This is what allowed you to import a flow definition from your public bucket to your NiFi canvas). You'll also notice that I have authorized by "nifiadmin" user identity read, write, delete on this bucket.  This allows my "nifiadmin" user identity to add or update (write) flow definitions into this bucket and delete flow definitions from this bucket.  From what you shared via the authorizations.xml, you have authorized "abc123" user identity read, write, delete on your public bucket.  So only thing to verify here is that "abc123" user identity is the exact same case sensitive user identity being displayed in the upper right corner of your NiFi UI when you are authenticated into your NiFi.  If they don't match then they are treated as different users.  It is the user identity shown in your NiFi that is being proxied and needs to be properly authorized in NiFi-Registry as above example.  (I know "abc123" is an example, but you get the idea here). Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more