@ppbirajdar    My next thought would be to just drop the cloned FlowFile right before the MergeContent since you intent is not to reassemble the exact same original file.  Your intent, if i understand correctly, is to just create one file that contains 1 FlowFile with each unique fragment index.   If that is the case, perhaps all you need is a detectDuplicate in front of your mergeContent that uses a combination of Fragment.identifier and fragment.index for determining if there is a duplicate FlowFile to drop.  You would simply set the "Cache Entry Identifier" to "${fragment.identifier}-${fragment.index}". Then only the first occurrence of a FlowFile with unique fragment.index+fragment.identifier will make it in to the connection queue for your MergeContent processor.  Then the Defragment merge strategy will work successfully giving you what I believe you are looking for here.   Hope this helps with your use case, Matt ... View more

@vivek12    Your last statement is confusing to me. Are you saying only the invokeHTTP processor is stopped after a NiFi restart?   This implies that someone stopped it or the flow.xml.gz is not getting updated with that processors last known state.  I'd inspect what is written in your flow.xml.gz on each node to make sure they all show the state as running. The last known state of a processor is not something that is checked as node join a cluster.  Nodes joining are checked to make sure their flow.xml matches the elected cluster flow and if it does, the node can join.  That node will then be told to start those components which are running in the cluster. When you restart the entire cluster each node presents its flow.  First node has Flow x and gets 1 vote, next node flow is checked and it matches exactly, that flow gets 2 votes.  Next node comes in and his flow is not exactly the same, so it gets 1 vote.  The flow with the most votes becomes the cluster flow.  If you have an even number of nodes (for example 4 nodes) and 2 node's flows get a vote and other 2 node's get a different vote.  since you have 2 vs 2, NiFi will end up picking one at random.  My concern here is that some node(s) have last known state as stopped while another is running.  So sometimes with a complete restart of your cluster you end up starting the flow with the stopped state on this processor. The other possibility is this invokeHTTP processor is failing validation on some node on startup and resulting in processor being stopped. Have you tried copying the flow.xml.gz from one node to all the other nodes?   Hope this helps, Matt ... View more

@SamCloud    The intent of the detect duplicate is to allow the first FlowFile through and then route all other FlowFiles with same cache entry identifier to duplicate.  So when a FlowFile reaches this processor it checks the Distributed Map Cache for the Cache Entry identifier, if it does not exist it is added to the cache and the FlowFile is routed to non-duplicate.  If it is found, then FlowFile is routed to duplicate. Based on the description you provided, it is working as intended/designed. Hope this helps, Matt ... View more

@psilvarochagome    The new product repositories are accessible only with a valid credentials.  Within HDF, the HDF 3.5.x + releases are the he first to require account access to download the binaries. You should reach out to your Cloudera Account team if you have trouble with your credentials.   Hope this helps, Matt ... View more

@ppbirajdar    If you have multiple FlowFiles with the same fragement.index and fragment.identifier, this indicates that at some point in your dataflow this FlowFIle was likely cloned (Same FlowFile routed to multiple relationships) or split (two FlowFiles created from one).  For example you dragged the success relationship twice.  Each of those FlowFiles will have however different UUID assigned to them.  You could use NiFi Provenance to look at the lineage of these FlowFIles to see at what point in you flow the clone was created.  In your case a CSV record that gets split into two FlowFIles (one with invalid records and another with some valid records) is going to be your issue since you route them to same destination processor. You didn't share your MergeContent processor configuration, but keep in mind the following: 1. A processor executes based on its configured run Schedule.  The default run schedule ins always 0 secs which means execute as fast as it can. This means a processor will execute and once it completes that execution, it will execute again. (If at execution there is no work to do, it will yield for a period of time before next execution is allowed to avoid excessive CPU usage when no work exists). 2. At time of Execution the MergeContent will look at what new FlowFiles since last execution and allocate those FlowFiles to bins.  If at end of that execution a bin meets the minimums configured, the bin will be merged (Mins don't matter if using Defragment merge strategy).  3. When using Defragment, the expectation is that all FlowFiles with same Fragment.identifier also have same Fragment.count and that only one FlowFile exists in a bin for each fragment.index value. It would be impossible to reassemble to original FlowFile since you have multiple FlowFiles now with the same fragment.identifier and fragment.index values.  The real question is whether you are actually trying to reassemble the exact original FlowFile or simply create a single FlowFile for which to trigger downstream processing? If the latter is the case, don't use Defragment merge strategy.  Use Bin-Packing Algorithm and set fragment.identifier as the "Correlation Attribute Name".  Since you really have no idea how many fragments there will be at this point, set your mins number of entires to a value large enough it would never be reached for a single fragment.identifier and then set the "max bin age" to a value high enough that you fee comfortable all fragments would have made it to the MergeContent.  Max.bin.age will force a bin to be merged at x amount of time after first FlowFile was allocated to that bin even if mins were not met. Also make sure you have configured n+1 number of bins, where "n" is the number of expected unique fragment.identifiers you expect to have at any one being binned by the MergeContent processor. Hope this helps, Matt ... View more

@abhinav_joshi  This is something that only affects the Apache NiFi releases only. The removal of some nars is documented in the Apache NiFi migration guidance wiki here: https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance   "Starting in 1.10,  the following nars were removed from the default convenience binary.  These include kite-nar, kafka-0-8-nar, flume-nar, media-nar, druid-controller-service-api-nar, druid-nar, other-graph-services-nar.  You can still get them from the various artifact repositories and use them in your flows but we cannot bundle them due to space limitations by default." For Apache NiFi users, I recommend that users create a custom NiFi lib directory by adding the following property to their nifi.properties file:   nifi.nar.library.directory.<unique-string>=/<path to>/custom-lib1 for exmaple: nifi.nar.library.directory.NiFi-1-10-nars=/nars/nifi-1-10-lib   You can add as many custom lib directories as you want.  Then place any nars that are noted as removed via the migration guidance documentation into one of these custom lib paths before upgrading.  Then as part of your upgrade process add the above property to your nifi.properties file. Then later if additional nar bundles are deprecated, you can create another custom lib dir or just add those nars to your existing custom lib directory before upgrading.  This allows you to avoid the downtime you encountered.   The Cloudera releases of HDF and CFM include are built with all the nars by default currently as they are not affected by space limitations. Hope this helps, Matt ... View more

@Rohitravi  The None of NiFi's processors will release any FlowFiles to a downstream connection until the end of the thread operation.  This is to protect users from dataloss and in some cases data duplication in the result of a failure. In the case of a SplitText processor you have configured to split on every 10 lines.  The processor will stream the content of the first 10 lines in to a content claim in the content_repository and create a new FlowFile record pointing at that claim.  The next 10 lines may or may not go into that same content claim and another FlowFile record is created. above process continues until all splits have been created.  Then The processor releases all FlowFile created to the downstream connection at the same time. NiFi does not guarantee FlowFile processing order. You can adding the FirstInFirstOutPrioritizer to the downstream connections to help with ordering some. Hope this helps, Matt ... View more

@krishnaraj_v13    The error output is telling you that your NiFi node(s) have not been granted the proxy policy in your NiFi Registry.  Your NiFi-Registry policies are managed locally within NiFi-Registry. Your NiFi is setup to use Ranger ti handle authorizations and i see you mentioned you granted your NiFi nodes /proxy in Ranger, but these policies only apply for NiFi and not NiFi-Registry. Based on the authorizers.xml shared from NiFi-Registry, I can see you defined your NiFi nodes as local users in the file-user-group-provider, but did not also configure those nodes in the file-access-provider.  The file-access-provider actually created the initial policies in the authorizations.xml file and assigns users to those created policies. Note: Both NiFi and NiFi-Registry will only create the users.xml and authorizations.xml files if they do not already exist.  So modifications to these providers in the authorizers.xml file will not result in modifications to these files if they already exist. To resolve the error you are seeing you need to login to your NiFi-Registry with your initial admin user and grant your NiFi nodes the the following policies: 1. "Can proxy user requests".  (solves current error) 2. "Can Manage buckets" --> Read.  (allows NiFi nodes to read buckets to see if new flow versions have been committed) Hope this helps, Matt ... View more

@vivek12    The last known state of the components is written to the flow.xml.gz file. Make sure the NiFi service user owns and proper permissions on this file. Do you have multiple nodes in your NiFi cluster?  If so, make sure that property is set to true in every node.  All it takes is one node to have it false to cause issues. What version of Apache NiFi are you running? Thanks, Matt ... View more

@guoloi    So your current authorizers.xml file uses these elements: managed-authorizer ---(references)--> file-access-provider (builds authorizations.xml file). ---(references)---> ldap-user-group-provider (syncs users and groups from ldap). The problem you have here is your NiFi nodes are not being synced from your ldap.  So the authorizer knows nothing about these clients/users.  In a NiFi cluster, the NiFi nodes also must be authorized for a variety of policies.   This means that you need to change the configuration of your authorizers.xml so that is supports multiple user-group-providers. managed-authorizer --- (references) --> file-access-provider (builds authorizations.xml file). --- (references) ---> composite-user-group-provider (allows user to reference multiple user-group-providers) ---(references both)---> ldap-user-group-provider (syncs users and groups from ldap) ---(and)--- file-user-group-provider (supports locally added users and groups).   The composite-user-group-provider looks like this: <userGroupProvider> <identifier>composite-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider>  and your file-user-group-provider looks like this: <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">CN=confluent, OU=NIFI</property> </userGroupProvider>   The file-user-group-provider will build the users.xml file which will hold all your "local" users and groups not being synced from your ldap.  The provider will add the local user/client " CN=confluent, OU=NIFI" when it builds the users.xml for the first time. The next issue that you will run in to is that the user "CN=confluent, OU=NIFI" is not authorized for the /proxy policy in the authorizations.xml file which you already have in place.  Normally you would have configured the file-access-policy-provider which builds the authorizations.xml file with this user also, but setting this now will not update the authorizations.xml file with any new entries (only created from config file if it does not exist already).   Its configuration would have looked like this: <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">guoloi</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=confluent, OU=NIFI</property> <property name="Node Group"></property> </accessPolicyProvider>   So two multiple options here: 1. Remove the authorizations.xml file from every NiFi node so it is recreated on startup.  Downside is new authorizations.xml file will only have policies set for the users "guoloi" and "CN=confluent, OU=NIFI".  any other policies you setup later yourself via the NiFi UI will need to be setup again. 2. Manually edit the authorizations.xml file ( you will need to start NiFi once first so users.xml file is created.  Then you can get the UUID assigned to the user "CN=confluent, OU=NIFI" from that file.  The UUID is what the /proxy policy would be assigned to.  Adding lines like this: <policy identifier="d59a54f7-6dd6-34ad-a279-a26ffdb9eef8" resource="/proxy" action="R"> <user identifier="38e35829-435d-3be4-83b6-784cb560e855"/> </policy> <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W"> <user identifier="38e35829-435d-3be4-83b6-784cb560e855"/> </policy> <policy identifier="6dbdbffd-8a7d-32e1-ba3e-f600e6c69791" resource="/proxy" action="D"> <user identifier="38e35829-435d-3be4-83b6-784cb560e855"/> </policy> note: substitute your UUID from users.xml in place of "38e35829-435d-3be4-83b6-784cb560e855" in above. 3. Startup only one NiFi node and access the UI.  Go to global menu in upper right corner and select "Users" to verify user "CN=confluent, OU=NIFI" exists.  Then go to global menu and select policies.  Form list of policies locate "proxy user requests" and add "CN=confluent, OU=NIFI" to that policy.   Then copy the authorizations.xml file from this NiFi node to your other NiFi nodes and start them.   Hope this helps, Matt   ... View more

@vivek12    If you NiFi dataflows are all coming up stopped after a NiFi restart, this indicates you have the following property in your nifi.properties file set to false: nifi.flowcontroller.autoResumeState=   Make sure that this property is set to true so that all components return to last known state from before NiFi was last shutdown.   Hope this helps, Matt ... View more

@NY    Anything you can do via the NiFi canvas, you can also accomplish via the rest-api. So you can create a queue listing via a rest-api call: curl http://<nifi-hostname>:<port>/nifi-api/flowfile-queues/<connection-uuid>/listing-requests The above call will return a response that includes the URL you need to use to retrieve those results. for example: http://<nifi-hostname>:<port>/nifi-api/flowfile-queues/<connection-uuid>/listing-requests/1d98d557-0171-1000-ffff-ffffd559ca47 Then query for the listings results curl http://<nifi-hostname>:<port>/nifi-api/flowfile-queues/<connection-uuid>/listing-requests/1d98d557-0171-1000-ffff-ffffd559ca47 This return a json with all the FlowFiles from that specific connection queue from all nodes in your NiFi cluster.  That json would need to be parsed for info like nodes where the "lineageDuration" epoch time is x amount of time older then now, the "clusterNodeAddress" (which node holds the file), and maybe filename". and then delete the queue listing when done. (this is important or it stays around using heap space). curl -X DELETE http://<nifi-hostname>:<port>/nifi-api/flowfile-queues/<connection-uuid>/listing-requests/1d98d557-0171-1000-ffff-ffffd559ca47 Hope this helps, Matt ... View more

@mary_b    You could use a Java Regular expression like this: .*_(20)\d\d(0[1-9]|1[012])(0[1-9]|[12][0-9]|3[01]).txt   The above will match everything before "_" and then must match a number that starts with "20", followed by  to digits "\d\d" (this matches any year in 2001-2099), followed by  "(0[1-9]|1[012])" (this matches months 01-09 or 10,11,12), followed by "(0[1-9]|[12][0-9]|3[01])" (this matches days 01-09 or 10-29 or 30 or 31), ending with the text string ".txt".   There are some good regex testers available on the web you may find useful. https://regexr.com/ http://myregexp.com/ https://www.freeformatter.com/java-regex-tester.html   Hope this helps, Matt   ... View more

@Gubbi    The Credentials File you created should be owned and only accessible by the NiFi service user.  Since all components in NiFi are executed by the NiFi service user, technically anyone who has access to the NiFi UI and knows where this credentials file exists on the server would be able to use it in a new processor they add to canvas. Above being said, you should be setting up proper authorization policies on your NiFi to limit what users can actually view the configuration of your putS3Object processor.  If a user cannot view its configuration, that user will not know the bucket or location of your credentials file on disk.  Or you configure the access and secret key directly in processor so it never in a file on disk where unauthorized users could find it and use it.   You may also consider using the AWSCredentialsProviderControllerService as well and limited access to that controller service to only select users.  Then if others unauthorized users add S3 processors to the canvas they would not be able to select that AWSCredentialsProviderControllerService.   You can further protect  your access key and secret key by not using the credentials file and instead enter  those values directly in the corresponding properties in the processor or AWS controller service.  Then they do not exist on disk anywhere for a user to find and use. The advantage to using the AWS controller service is that you can still give users access to the processor but block all but specific users access to the AWS controller service.  This allows users to operate the processor, look at data provenance for that processor, etc while still not having access to view the secret and access keys in the referenced AWS controller service.   Hope this helps, Matt ... View more

@Umakanth    The "Unable to find valid certificate path to requested target" suggests an incomplete certificate trust chain.   I would suggest looking getting a verbose output for your NiFi's truststore.jks and keystore.jks files. keystore -v -list -keystore <truststore.jks or keystore.jks> If this is a NiFi cluster, make sure you check the truststore.jks on all your NiFi nodes, they need to be able to successfully trust each other as well.   When you access the UI via any node's URL, that request is sent to the cluster coordinator and replicated to all nodes in your cluster.  Those node to node communication with also use mutual TLS authentication.  A complete trust chain will include all Certificate Authority (CA)s (intermediate(s) all the way up to the root CA).  When you look at your verbose output for your keystore.jks (it must contain only 1 PrivateKeyEntry that supports both clientAuth and ServerAuth) you will see an owner and issuer for your PrivateKeyEntry.   That Issuer is the signer of your certificate.   The signer may be an intermediate CA meaning that its not both the issuer and owner its certificate.  If that CA has a different issuer, then it was signed by another CA.  This means your truststore.jks must also include that CA as well.  This process repeats until you reach the root CA.  The rootCA will have the same DN for both owner and issuer.  Once all those TrustedCertEntries exist in your truststore, you have the complete trust chain.  The same holds true for the client certificate you are using to connect to your NiFi cluster.  I noticed it is signed/issued by "localhost".  Make sure localhost trustedCertEntry also exists in all your NiFi node's truststore.jks.  If the localhost CA was signed by another CA, make sure that you have that traced all the way back to the root CA as well.   Another thing you may want to try to see if it is a node to node SSL issue or a client to node issue, is start only one node in your NiFi cluster and try to access it after it is fully up.   Hope this helps, Matt ... View more

@saivenkatg55    Sorry to hear you are having space issues with your content repository.  The most common reason for space issues is because there are still active FlowFiles referencing the content claims.  Since a content claim cannot be moved to archive sub-directory or deleted until there are no FlowFiles referencing that claim, even a small FlowFile can still queued somewhere within a dataflow can result in a large claim not being able to be removed.  I recommend using the NiFi Summary UI (Global menu --> Summary) to locate connections with flowfiles just sitting in them not getting processed.  Look at the connections tab and click on "queue" to sort connections based on queued FlowFiles.  A connection with queued FlowFiles, but shows 0 for both "In/Size" and "Out/Size" are what I would be looking for.  This indicates in the last 5 minutes that queue has not changed in amount of queued FlowFiles.  You can use the got to arrow to far right to jump to that connection on the canvas.  If that data is not needed (just left over in some non active dataflow), right click on the connection to empty the queue.  See if after clearing some queues the content repo usage drops. Is is also possible that not enough file handles exist for your NiFi service user making clean-up not to work efficiently.  I recommend increasing the open files limit and process limits for your NiFi service user.  Check to see if your flowfile_repository is large or if you have content claims moved to archive sub-directories that have not yet been purged.  Does restart of NiFi which would release file handles trigger some cleanup of the repo(s) on startup? It is also dangerous to have all your NiFi repos co-located on the same disk for risk of corruption to your flowfile repository which can lead to data loss.  The flowfile_repository should always be on its own disk, the content_repository should be on its own disk, and the provenance_repository should be on its own disk.  The database repository can exist on a disk used for other NiFi files (config files, local state, etc.) https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-performance/ta-p/244999 Here are some additional articles that may help you: https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418 https://community.cloudera.com/t5/Community-Articles/How-to-determine-which-FlowFiles-are-associated-to-the-same/ta-p/249185   Hope this helps, Matt           ... View more

@sfishman    You are correct that NiFi does not support multiple private keys within a keyring.  I encourage you to create an Apache NiFi Jira with your details and this enhancement request. https://issues.apache.org/jira/browse/NIFI   Matt ... View more

@sfishman    The EncryptContent processor supports encryption/decryption using the PGP encryption Algorithm. PGP requires that the relevant PGP properties have been configured. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.3/org.apache.nifi.processors.standard.EncryptContent/index.html   Hope this helps, Matt     ... View more