Hello, I have a Nifi SpringContextProcessor which is establishing connection to the Cassandra. On processor START it is establishing the connection, but on processor STOP it is not releasing the connection consistently. (Some time it release and some time it does not) Do you have any idea how I can handle this connection? Below is the piece of code used for connection - session = DseCluster.builder() .addContactPoint(cassandraUrl) .withPort(cassandraPort) .withSpeculativeExecutionPolicy( new ConstantSpeculativeExecutionPolicy( 500, // delay before a new execution is launched 2 // maximum number of executions ) ) .withAuthProvider(new DseGSSAPIAuthProvider("HTTP")) .build() .register(QueryLogger.builder() .withConstantThreshold(100) .withMaxQueryStringLength(20) .build()) .connect(cassandraKeySpace); Thanks Mahendra ... View more

Hello, I need to query a cassandra table from NiFi to get small result set. I am using 'QueryCassandra' processor , and it is working fine with 'username' & 'password' for authentication. But due to some reason I cant go ahead with username & password, instead I should use kerberos keytab to authenticate cassandra. 'QueryCassandra' processor has no option to provide keytab & principal for authentication. Is there any way to connect using kerberos Thanks, Mahendra ... View more

Hello, I have a scenario where nifi flow file is flowing which is of up to 5 MB. During the flow I need to get some info from Cassandra master table and use it for the flow file validation. If I go with 'QueryCassandra' that fetches result into flow file content, so I will loose my original source flow file content ! Cassandra table result will be of only 2-3 rows (less in size), so is there any way to get this result into an attribute and retain my original flow file content? One option I have is SpringContext processor, Is there any other better approach ? Any suggestion will be appreciated. Thanks, Mahendra ... View more

Hello, Seeing below error in nifi kafka consumer frequently - "org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member" We are using nifi 1.5 Could any one please let me know the solution for this? Thanks, Mahendra ... View more

Hello, I have a json input for which I need to add one key-value("correlationId": "123456789") under 'tripSummaryMetaData' section dynamically. I tried with below spec and its working fine, but the issues is when 'tripSummaryMetaData' is not available the it is creating new 'tripSummaryMetaData' and adding my attribute. When 'tripSummaryMetaData' is not available the it should not add my new attribute, is that possible in JOLT? Thanks in advance. Input: { "tripSummaryMetaData": { "orgId": "demo", "message_type_id": "MB-CD-MSG0001" }, "eventSummary": { "tripId": "2018072204cc7851d3e84f7ea8a1155c95f65b06", "referenceData": "050bfa62-430a-45a5-b7a6-78d688d45167", "gpsData": [{ "gpsTimeReceived": 1532319157338, "gpsTime": "2018-07-22T21:12:38-07:00" }] } } Output: { "tripSummaryMetaData" : { "orgId" : "demo", "message_type_id" : "MB-CD-MSG0001", "correlationId" : "123456789" }, "eventSummary" : { "tripId" : "2018072204cc7851d3e84f7ea8a1155c95f65b06", "referenceData" : "050bfa62-430a-45a5-b7a6-78d688d45167", "gpsData" : [ { "gpsTimeReceived" : 1532319157338, "gpsTime" : "2018-07-22T21:12:38-07:00" } ] } } Spec : [{ "operation": "default", "spec": { "tripSummaryMetaData": { "correlationId": "123456789" } } }] Thanks, Mahendra ... View more

Hello, I am trying to write the streaming data to Hive table using 'PutHiveStreaming' processor. Processor is working fine, but only problem is its too slow in writing data to the table. I have cluster with 3 nodes, where I tried running processor with 'All nodes' and '50 Concurrent threads' , it is able to write maximum of 3-4 flowfiles per second !!! Attached is the screen shot of configurations. hivestreaming.png Thanks, Mahendra ... View more

Hello, We have cluster with 3 nifi nodes in it. One of the node is getting disconnected once in a while. Issue get resolved temporarily when we restart the node. Error - node-down.png Thanks, mahendra ... View more

hello, Facing an issue with 'InvokeHTTP' NiFi processor. Have configured below URL for GET operation - https://******/**/users/${userId} When I run the processor some time it will get proper response from url. But if I restart the processor, suddenly it will start telling 'URL not found' error and route the flow file to 'No Retry' relationship. If I try to restart the processor 2-3 times then once it will start responding correctly. Please suggest me to resolve the issue. screen shots of config - Thanks, Mahendra ... View more

Hello, I am trying to use 'ListHDFS' processor and it is giving error and not listing the files. I have configured DistributedMapCacheServer & Client and enabled it. Please find the below error screen shot - And when I try to right click on the processor and 'view state' it is giving below error : Thanks in advance ! Mahendra ... View more

Hello, I am trying to use ListHDFS processor and in environment where I have a cluster with 3 nifi nodes. I am trying to run this processor only on Primary node. Configured DistributedMapCacheClientService & DistributedMapCache for this processor but still itis giving below error , any idea what could be the reason ? ERROR [Timer-Driven Process Thread-1] o.apache.nifi.processors.hadoop.ListHDFS ListHDFS[id=d63638b0-0161-1000-0000-00002a87ba72] Failed to retrieve timestamp of last listing from Distributed Cache Service. Will not perform listing until this is accomplished. Thanks in advance ! Mahendra ... View more

Hello All, Any idea what could be the reason to get this below error in NiFi app logs ? Also I am seeing same error on NiFi UI as well. ERROR : p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre} 2018-01-24 07:07:10,663 WARN [StandardProcessScheduler Thread-2-SendThread(localhost:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. 2018-01-24 07:07:11,764 WARN [StandardProcessScheduler Thread-2-SendThread(localhost:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. 2018-01-24 07:07:11,865 WARN [StandardProcessScheduler Thread-2] o.a.h.h.zookeeper.RecoverableZooKeeper Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.h.h.zookeeper.RecoverableZooKeeper ZooKeeper exists failed after 4 attempts 2018-01-24 07:07:11,865 WARN [StandardProcessScheduler Thread-2] org.apache.hadoop.hbase.zookeeper.ZKUtil hconnection-0x6aa48200x0, quorum=localhost:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:188) at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:408) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.h.hbase.zookeeper.ZooKeeperWatcher hconnection-0x6aa48200x0, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:188) at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:408) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.h.h.c.ConnectionManager$HConnectionImplementation Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.n.c.s.StandardControllerServiceNode HBase_1_1_2_ClientService[id=5a504684-9c3a-10ae-ffff-ffffe9e6e06b] Failed to invoke @OnEnabled method due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 24 07:07:11 EST 2018, RpcRetryingCaller{globalStartTime=1516795614116, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase : {} org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 24 07:07:11 EST 2018, RpcRetryingCaller{globalStartTime=1516795614116, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:188) at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:408) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ... 19 common frames omitted Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:906) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) ... 23 common frames omitted Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895) ... 26 common frames omitted 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of HBase_1_1_2_ClientService[id=5a504684-9c3a-10ae-ffff-ffffe9e6e06b] due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 24 07:07:11 EST 2018, RpcRetryingCaller{globalStartTime=1516795614116, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase ... View more

Hi, I am trying to convert UTF-8 character set to ASCII using ConvertCharacterSet and I am not getting desired output. Input used - %7Btest%7B Processor configured with input as UTF-8 and output configured as ASCII. Any extra configuration required? Please help Thanks, Mahendra ... View more