@Wesley Bohannon The issue you are most likely running in to is caused by only having 1 bin. https://issues.apache.org/jira/browse/NIFI-4299 Change number of bins to at least 2 and see if the resolves your issue. Thanks, Matt ... View more

@Hadoop User I am unfortunately not a Hive or Hadoop Guru. Both errors above are being thrown by the Hive and Hadoop client libraries that these processors use and not by NiFi itself. Hopefully the above log lines are followed by full stack traces in the nifi-app.log. If not, try enabling DEBUG logging to see fi you can get a stack trace output. That stack trace may provide the necessary details to help diagnose what is causing the issue. Hopefully a Hive or Hadoop Guru will then be able to provide some assistance here. I also suggest providing the details for the HiveConnectionPool controller service you setup that is being used by this PutHiveQl processor. Thanks, Matt ... View more

@Timothy Spann I am not sure what you mean by "Not updating". Also what Provenance implementation are you using (Persistent, WriteAhead, or Volatile)? ... View more

@J. D. Bacolod Those processors were added for specific uses cases such as yours. You can accomplish the same thing almost using the putDistributedMapCache and FetchDistributeMapCache processors along with an UpdateAttribute processor. I used the UpdateAttribute processor to set a unique value in a new attribute named "release-value". In my case the value is assigned it was: The FetchDistributedMapCache processor then acts as the wait processor did looping FlowFile in the "not-found" relationship until the corresponding value is found in the cache. The "release-value" is written to the cache using the PutDistributedMapCache processor down the other path after the InvokeHTTP processor. It will receive the "Response" relationship. Keep in mind, the FetchDistributedMapCache processor does not have an "expire" relationship. If a response if never received for some FlowFile or the cache expired/evicted the needed value, those FlowFiles will loop forever. You can solve this two ways: 1. Set File Expiration on the connection containing the"not-found" relationship that will purge files that have not found a matching key value in the cache by the time the FlowFile's age has reached x value. With this option aged data is just lost. 2. Build a FlowFile expire loop which kicks these looping not-found FlowFiles out of loop after x amount of time so they can be handled by other processors. This can be done using the "Advanced" UI of an UpdateAttribute processor and a RouteOnAttribute processor: The UpdateAttribute sets a new attribute I called "initial-date" if and only if it has not already been set on the FlowFile. This can be done as follows using the "Advanced" UI of the UpdateAttribute processor : The RouteOnAttribute Processor then compares the current date plus x milliseconds to that attribute's value to see if file has been looping for more the x amount of time. (Using 6 minutes (360000 ms) as an example, my RouteOnAttribute would have a property/routing rule like this: FlowFiles that have been looping for 360000 milliseconds or more will then get routed to "expired" relationship where you can choose what you want to do with them. As you can see the processors wrap the above flow up in only two processors versus 5 processors you would need in older versions to get same functionality. Thanks, Matt ... View more

@Narasimma varman In order to access a secured NiFi's UI, successful user authentication and authorization must occur. In HDF a NiFI CA is installed that takes care of building valid keystores and truststores for your NiFi nodes, but it does not create user certificates for you. Typically the above error indicates NiFi did not trust the client certificate it was passed or a client certificate was not passed at all. I would suggest staring by getting verbose outputs of your NiFi keystore.jks, truststore.jks, and users keystore.p12. The verbose output for each of these can be obtained using keytool. ./keytool -v --list -keystore <jks or p12 keystore file> In your Keystore.jks used by the NiFi server, you will see a single entry with two certificates included in it: Specifically you are looking for the "PrivateKeyEntry". This PrivateKeyEntry will show a user DN (It will be in the form of CN=<server FQDN>, OU=NIFI). You will then see an issuer line which will also have a DN for the NiFi CA. This PrivateKeyEntry should have an extended key usage that allows the key to be used for both client auth and server auth. Something else (not related to your issue) i noticed was your browser URL is "localhost". The NIFi CA will generate a server certificate based off the hostname of the server and not localhost. This will require you to add an exception in your browser at some point. (This is because the cert passed to your browser from your NiFi server will say it belongs to server XYZ, but your browsers knows it was trying to connect to localhost. So it appears as a man in the middle type attack (some en-point using another end-points cert). In your truststore.jks used on your NiFi servers, you will see a single certificate. It will be a "TrustedCertEntry" for the NiFi CA. The truststore.jks file can contain 1 to many trusted cert entries. Each trusted cert entry is derived from the public key of a CA or self-signed cert. When a client (user or another server) negotiates a connection with the server a TLS handshake occurs. As part of this negotiation, the server expects to receive a client certificate which it can trust. If a trusted client cert is not received, the connection is typically closed by the server. Your client keystore.p12 file will also need to contain a PrivateKeyEntry. In TLS negotiation that occurs with the server, the DN associated to that PrivateKeyEntry is passed to the server. If that certificate was self-signed, the truststore would need to contain the public key for that certificate as a TrustedCertEntry before that certificate will be accepted for authentication. Beyond authentication is authorization, but it does not appear you are getting that far yet. Thanks, Matt ... View more

@Hadoop User Please start a new question rather then asking multiple unrelated questions in a single post. This makes it easier for community users to find similar issues. It also help other members identify unanswered questions so they may address them. This question would likely go unnoticed otherwise. I would need to do some investigation to come up with a good solution, but other community members may have already handled this exact scenario. By starting a new question, all members following the "data-processing" or "nifi-processor" or "nifi-streaming" will get notified of your question. Thanks, Matt ... View more

@Hadoop User The ExtractText processor will extract the text that matches your regex and assign it to an attribute matching the property name on the FlowFile. The content of the FlowFile remains unchanged. Then you update a FlowFiles Attribute and finally use PutHDFS to write the content (which at this time you have not changed at all) to HDFS. If your intent is to write the modified string to HDFS, you need to update the actual content of the FlowFile and nit just create and modify attributes. For that use case, you would want to use ReplaceText processor instead. You would configure ReplaceText similar to the following: The above will result in the actual content of the FlowFile being changed to: [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] Thanks, Matt ... View more

@Foivos A The banner is a NiFi core feature and is not tied in anyway to the dataflows you select or build on your canvas. You are correct that the best approach for identifying which dataflows on a single canvas are designated dev, test, or production is through the use of "labels". In a secure NiFi setup, you can use NiFi granular multi-tenancy user authorization to control what components a user can interact with an view. If you use labels, you should set a policy allowing all user to view that specific component, so even if they are not authorized to access the labeled components, they will be able to see why via the label text. Thanks, Matt ... View more