@sho aa Based on your first screenshot provided, it appears the putHbaseJson processor is processing FlowFiles. The stats indicate that ~783,000 FlowFiles had been pulled from the inbound connection queue in the last 5 minutes. The PutHbaseJson processor also shows that it has an active thread (1) at the time of the screenshot. It appears you are auto-terminating both the success and failure outbound relationships on your PutHbaseJson processor. If FlowFiles were auto-terminating on the Failire relationship, you should be seeing a Red square (bulletin) which your screenshot does not show. The bulletin reports an error log was produced. So I can only assume if you are not seeing bulletins being produced that FlowFiles are being auto-terminated via the success relationship. Have you verified that nothing is written to hbase? Have you verified all your hbase configurations are correct? The flow above indicates the issue is more of a throughput problem. The SplitJSON processor is producing large batches of output FlowFiles at one time (that is how that processor works). Back pressure is then applied on the connection between SplitJSON and PutHbaseJson processors (connection highlights red to indicate back pressure is being applied). SplitJSON will not be allowed to execute again until that back pressure drops below the configured threshold. The PutHBaseJson processor continues execute against FlowFiles in the connection queue. Have you tried the following to see if you see an improvement in throughput of the PutHbaseJson processor: 1. Increase configured batch size. The default is 25. Keep in mind that the content of every FlowFile in a batch is held in heap memory. Setting this value too high can lead to Out Of Memory (OOM) condition in NiFi's JVM. So increment slowly. 2. Increase configured concurrent tasks. Default is 1. One concurrent task allows this processor to execute on 1 batch at a time. Until that batch put is complete the next batch cannot be processed. Adding additional concurrent tasks will allow this processor to execute on multiple batches of FlowFiles at the same time. Keep in mind that you system does have a limited number fo cores. Setting this value too high can lead to your CPU becoming saturated. The max threads all processors on canvas must share is set via "Controller Settings" found in the global menu in the upper right corner of the NiFi UI. General rule of thumb is to set the "Maximum Timer Driven Thread Count" to a value equal to 2 to 4 times the number of cores on your NiFi server. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Josh Nicholson When using the kerberos-provider via the login-identity-providers.xml file. The user's full kerberos principal is going to be used every time. You can ignore the "default realm" property in the kerberos-provider provider (NiFi's code does not actually use it right now --> https://jira.apache.org/jira/browse/NIFI-6224 ) So when a user enters a username that does not include the "@<realm>" portion, the default realm as configured in the krb5.conf file configured in the nifi.properties file is used. That full DN is then passed through your configured identity.mapping.patterns. This means you need to have a pattern that matches on: ^(.*?)@(.*?)$ And a resulting value of: $1 so that only the username portion is then passed on to your configured authorizer. In the case of some user coming in with just username and other with full principal names... Those user coming in with just usernames must not being authenticated using the login provider. Even with a login provider configured the default TLS/SSL authentication is attempted first. So if these users have a trusted client certificate loaded in their browser it will be presented for authentication to your NiFi and those user will never see the login window. From a user certificate the full DN will be used to identify the user. That full DN is likely matching on your existing mapping pattern resulting in just the username you are seeing. So it is important that you not remove this existing mapping pattern, but instead add a second. nifi.security.identity.mapping.pattern.<any string> nifi.security.identity.mapping.value.<any string> Patterns are searched in a alpha-numeric order. First matching regex will be applied. Thank you, Matt ... View more

@Soumya Ghosh 1. The DistibuteMapCacheServer (DMC server) has not direct linkage to a dataflow built in NiFi. Thus when creating a template of your flow from NiFi for the purpose of generating your MiNiFi yaml file, the DMC Server will not be included in the template. Only the DMC client service will be included. I have not tried to manually add the DMC server to the yaml file, but is likely possible. MiNiFi does not have a rest-api where you can send command to add additional components like you can do it NiFi. 2. The only way to host a DMC server is via a NIFi instance. There is not way to execute a NAR file. 3. A major limitation to using the DMC server is lack of HA. If the NiFi instance hosting the DMC server crashes you lost all your cached data (assuming crash is not recoverable. The recommended path is to configure you ListFile processor to use one of the other available external cache server that offer HA capability. These alternate cache service are setup independent of NiFi or MiNiFi and will offer you what you need to support using Entity tracking in your ListFile processor running on MiNiFi. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more