Hello, thanks everyone for the prompt response. With some aid I was able to figure it out Mostly my problem was to understand the difference between the Grouping Regular Expression and extracting the date parameter which in my case are pretty much the same expression. Also I have to admit that the RouteText.Group attribute was not something easy to find even in the documentation. I feel that reading a TCP connection with logs and store it partitioned directly to a Hive table should be a fairly common use case, so I'm attaching the template as a grain of sand contribution. recordtexttopartition.xml Thanks again ... View more

@Carrick That property does support Expression Language, but you need to understand what this property is doing with the EL you provided. ${trigger.type} <-- This is an expression language statement. It is asking NiFi to check for the existence of an attribute on the incoming FlowFile, assigned to a Process group, found in the NiFi registry file, found in a JVM property and finally found in NiFi user env variable. (check in that order, first match ends search.) If not found nothing is returned. If found the value assigned to that attribute is returned. So in your case what is being returned is 'mouse' or 'keyboard'. So now the merge content processor is using mouse or keyboard as the correlation attribute name. And in you case there is no attribute named mouse or keyboard so all the flowfiles with null values are ending up in the same bin. As soon as you removed the EL and simply defined the attribute to use for correlation it worked. Thanks, Matt ... View more

Thank you @mclark @Timothy Spann @Andrew Grande. I eventually only needed to save and import the old templates. I forgot to mention that the HDF 2.0 had already been installed on the new server. ... View more

@mclark @Matt Burgess so in any case does the file needs to be read in to memory before it splits.?? either by lines or by bytes. i was hoping it starts the next process work once it receive first split.? in my case it waited 8 minutes until it split the 10GB file into 1200+ splits. If my files are about 100 GB each (I have 18 such files) I am scared to run the whole flow for all files. I may have to run for each file one by one.? ... View more

@Juthika Shenoy This error indicates and authorization issue. This is separate from authentication. I would start by looking at your nifi-user.log and see what DN is successfully authenticating by being denied authorization. Then verify that DN is included along with your node identity(s) in the users.xml file. If it is not, then that is your problem. I noticed from your post above you never provided an "Initial Admin Identity" in your authorizers.xml file. This is a must in order to get an initial admin added to the system so that that initial admin can then add additional users via the UI. You can take your user DN from the nifi-user.log and add it to your authorizers.xml file: <property name="Initial Admin Identity">Add user DN Here</property> Also make sure you still have your Node Identity(s) set in the authorizers.xml file as well. <property name="Node Identity 1">DN From Node 1 Cert Here</property> <property name="Node Identity 2">DN From Node 2 Cert Here</property> etc..... If every Node is using the same cert. That cert must have a Subject Alternative Name (SAN) entry for each nodes FQDN. From a security standpoint, it is not recommended using one cert for multiple servers. Finally, you will need to stop your NiFi nodes, delete your existing users.xml and authorizations.xml files form each of them, and then restart. NiFi will only create those tow files once. Once they have been created, changes to the authorizers.xml file will not trigger updates to them. Thanks, Matt ... View more

I didnt copy the flow.xml.gz but imported some templates which must have updated the flow.xml. Whats the fix to restart the NiFi instance? ... View more

@Josh Elser @srinivas padala The "Read/Write" stats on the processor have nothing to do with writing to your SQL end-point. This particular stat is all about reads from and writes to the NIFi content Repository. This helps identify where in your flow you may have disk high disk I/O in the form of either reads or more expensive writes. From the screenshot above, I see that this processor brought in off inbound connections 35,655 FlowFiles in the past 5 minutes. It read 20.87 MB of content from the content repository in that same timeframe. The processor then output 0 FlowFiles to any outbound connection (This indicates all files where either routed to a an auto-terminated relationship). Assuming only the "success" relationship was auto-terminated, all data was sent successfully. If the "failure" relationship (which should not be auto-terminated here) is routed to another processor, the 0 "out" indicates that in the past 5 minutes 0 files failed. The Tasks shows a cumulative total CPU usage reported over the past 5 minutes. A high "Time" value indicates a cpu intensive processor. Thanks, Matt ... View more

@mclark That certainly worked! Thank you so much. One additional question related to this, is there any other way for me to access the flowfile-content within the XSLT file which i would use through Transform XML? ... View more

@Matt Clarke , @boyer : I am also facing similar issue. In my case workflow is something like HandleHttpRequest -> InvokeHTTP -> output port -> goes to custom processor in different process group which has only one property setup -> input port -> HandleHttpResponse, I configured same StandardHttpContextMap in both HandleHttpRequest and HandleHttpResponse processors; how should I deal with this error with http.context.identifier in HandleHttpResponse processor? Any help is appreciated... ... View more

step1 : remove the authorization.xml from all the nodes rm /var/lib/nifi/conf/authorizations.xml step2 : update node identities in advanced_nifi_ambari_ssl_configuration section with <property name="Node Identity 1">CN=hostname1, OU=XXXXX</property> ... for all the nodes (make sure you remove comment tags in XML). step3 : restart the NiFi service found that authorizations.xml is not being update after first generation, hence this is causing the problem. ... View more