Member since
09-29-2016
54
Posts
9
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1587 | 07-03-2017 09:12 PM | |
1302 | 05-22-2017 08:19 PM |
10-30-2017
06:36 PM
@Xu Zhe, yes the cluster is fully stable now.
... View more
10-27-2017
11:47 AM
Hi @Xu Zhe As mentioned in the comments by Wynner, the solution I used was to clean up the NiFi repositories and restart the cluster nodes at the same time.
... View more
09-06-2017
03:38 PM
Hi @Matt Burgess, I was able to connect successfully following your guidelines. I had to call the elasticsearch folder from the X-Pack Zip for the ES version 5.0.1 along with the 5.0.1 Transport Jar in the X-Pack Transport Location property as a comma-delimited list. Thanks for all the help, greatly appreciate it!
... View more
08-28-2017
07:16 PM
1 Kudo
Hello, I am trying to send feed from nifi version nifi-1.2.0.3.0.0.0-453 to elasticsearch version 5.5.1 however I keep getting the following error: PutElasticsearch5[id=f4da516d-2b44-1955-0000-00000a063fa3]
PutElasticsearch5[id=f4da516d-2b44-1955-0000-00000a063fa3] failed to process due
to org.apache.nifi.processor.exception.ProcessException:
org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be
loaded and/or configured; rolling back session:
org.apache.nifi.processor.exception.ProcessException:
org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be
loaded and/or configured
The full error log is the following: ERROR [Timer-Driven Process Thread-16] o.a.n.p.elasticsearch.PutElasticsearch5 PutElasticsearch5[id=f4da516d-2b44-1955-0000-00000a063fa3] Failed to create Elasticsearch client due to org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be loaded and/or configured: org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be loaded and/or configured
org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be loaded and/or configured
at org.apache.nifi.processors.elasticsearch.AbstractElasticsearch5TransportClientProcessor.getTransportClient(AbstractElasticsearch5TransportClientProcessor.java:222)
at org.apache.nifi.processors.elasticsearch.AbstractElasticsearch5TransportClientProcessor.createElasticsearchClient(AbstractElasticsearch5TransportClientProcessor.java:170)
at org.apache.nifi.processors.elasticsearch.AbstractElasticsearch5Processor.setup(AbstractElasticsearch5Processor.java:94)
at org.apache.nifi.processors.elasticsearch.PutElasticsearch5.onTrigger(PutElasticsearch5.java:175)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.nifi.processors.elasticsearch.AbstractElasticsearch5TransportClientProcessor.getTransportClient(AbstractElasticsearch5TransportClientProcessor.java:215)
... 15 common frames omitted
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.xpack.XPackPlugin
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:451)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:116)
at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:81)
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:106)
at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:228)
at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:69)
at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.<init>(PreBuiltXPackTransportClient.java:50)
at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.<init>(PreBuiltXPackTransportClient.java:46)
... 20 common frames omitted
I tried following the suggestions in the link: https://community.hortonworks.com/questions/81606/where-should-i-store-files-used-by-processors-in-a.html , but I am unable to fix the issue. I tried putting the x-pack jar and the transport jar for the elasticsearch version 5.5.1 on the X-Pack Transport Location configuration of the PutElasticsearch5 processor and I also tried using the x-pack jar and the transport jar of the elasticsearch version 5.0.1, however nothing seems to be working. Any suggestions on the issue would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
08-07-2017
08:54 PM
Hello, I am receiving around 300k-400k messages per day in NiFi version Nifi-1.2.0.3.0.0.0-453. The messages are coming as a Json and the goal is to put them into a hive table in near real time. I have built the following flow which is attached as an image in this post. The flow is working fine and doing everything like it is supposed to. The issue is that is not writing to putHiveStreaming fast enough and as a result messages start getting queued up in the flow. I have been playing around with the Transaction per Batch and Records per Transaction configuration properties from the putHiveStreaming but I seem to be unable to get the putHiveStreaming processor to write faster but it does not seem to be working and we are getting messages a lot faster than what we are writing. Is there a way to configure the putHiveStreaming processor so that it can handle this type of load of 300k-400k messages per day at around 3 to 4 messages per second fast enough? Any insight on this issue will be deeply appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
07-06-2017
08:12 PM
@Matt Clarke that seemed to fix the issue, thanks a lot!
... View more
07-05-2017
10:22 PM
Full error trace: 2017-07-05 17:04:12,336 WARN [Replicate Request Thread-8] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/process-groups/04a08abd-d360-3539-ad01-835bc54126b0 due to {}
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:611)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:822)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 12 common frames omitted
... View more
07-05-2017
09:42 PM
Hello, I am on the process of changing the nodes of our current nifi cluster for completely nifi dedicated nodes. I usually ran a 4 node nifi cluster and today I added 4 new nodes to that cluster. The main goal is to take out the old 4 nodes and to keep the new 4 nifi dedicated nodes in the cluster. All the nodes in the cluster are currently running on the nifi version nifi-1.2.0.3.0.0.0-453. I added the 4 new nodes successfully and gave them the proxy permissions and everything. I am able to log in to the new nodes and access the UI from them and they connected to the cluster without any issues. However whenever I try to access a specific process group from one of the 4 new nodes I keep getting the following error: This is only happening whenever I try to access that specific process group from any of the new nodes. If I access it from any of the old nodes no error happens and I can access the process group just fine. I looked at the logs from the new nodes and this is what I see: 2017-07-05 17:10:56,745 WARN [Replicate Request Thread-10] o.a.n.c.c.h.r.ThreadPoolRequestReplicator
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:611)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:822)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 12 common frames omitted
Does anyone have any ideas of what could be causing this? Is there a configuration that I have to change to avoid this error? or a permission that I have to give these nodes? Any insights would be extremely helpful.
... View more
Labels:
- Labels:
-
Apache NiFi
07-03-2017
09:12 PM
I was able to resolve the problem. The issue was that my kbr5.conf file that I called in the nifi.kerberos.krb5.file line in the nifi.properties file, was not configured properly. After properly configuring it and calling it in the nifi.properties file I was able to successfully send to kafka. I also had to use the PublishKafka_0_10 processor instead of the PutKafka processor since the version of kafka I was trying to connect to was 0.10.
... View more
07-03-2017
06:14 PM
@Shashank Chandhok
I did some changes to the jass.conf file but I am still getting the same error. This is the new jass.conf file:
... View more
07-03-2017
04:01 PM
Hello @Shashank Chandhok, I named the jass.conf file zookeeper-jaas.conf and the configuration is the following: The PutKafka configuration is the following one: The boostrap.conf line I added is the following one: The kafka version that we are trying to connect to is: 0.10.0
... View more
06-30-2017
10:00 PM
Hello, I am trying to send messages to a kerberos ebabled kafka using a standalone NiFi node version nifi-1.1.0.2.1.2.0-10. I followed the instructions listed in the following link: How to configure HDF1.2 to send to and get data from kerberized kafka I inputted the keytab and principal of the kerberized kafka and added the path with the appropriate added line in the bootstrap.conf file of my NiFi instance. However whenever I try sending a message with the PutKafka processor it gives an error saying: "Failed while waiting for acks from kafka". I verified the logs in the NiFi instance and this is what I see: 2017-06-30 15:41:07,166 WARN [kafka-producer-network-thread] org.apache.kafka.common.network.Selector Error in I/O with <serveraddress>
java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62)
at org.apache.kafka.common.network.Selector.poll(Selector.java:248)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
at java.lang.Thread.run(Thread.java:745)
2017-06-30 15:41:07,298 WARN [kafka-producer-network-thread] org.apache.kafka.common.network.Selector Error in I/O with <serveraddress>
java.io.EOFException: null
Does anyone have any ideas of what could be going on? Any insights in this would be really helpful!
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
06-30-2017
09:43 PM
@Wynner I had to try a couple of times but after a couple of tries of restarting the nodes at the same time the node was able to join the cluster. Thanks for the help!
... View more
06-29-2017
07:11 PM
@Wynner, no logs show of the node. In the zookeeper logs I logs for the other two nodes in the cluster but not for the one that is having problems.
... View more
06-29-2017
02:47 PM
@Wynner yes, all of my zookeeper instances are running, we use an external zookeeper not the NiFi embedded zookeeper and all of the instances have been running fine. The day this issue started to happen apparently one of the instances was having issues but since yesterday all of the instances have been working fine and all the services seem to be running but still the node keeps having an issue connecting to zookeeper. However the other two nodes seem to be just fine connecting to zookeeper and joining the cluster.
... View more
06-27-2017
10:08 PM
Hello, I have a 3 node cluster all using the NiFi version nifi-1.2.0.3.0.0.0-453. The cluster has been working fine for the last couple of weeks, however today all of a sudden one of the nodes disconnected from the cluster and won't join the cluster back. I checked the logs and the error I see is the following: ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-06-27 17:54:40,179 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The node itself appears to still be running but is disconnected from the cluster. I tried restarting it but the same error keeps appearing over and over again. The other two nodes in the cluster are working fine. Does anyone have any idea of what could be causing this sudden issue? Any insights would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
06-15-2017
02:06 PM
thanks for the clarification @Bryan Bende!
... View more
06-14-2017
04:30 PM
Hello, I tried upgrading NiFi from version 1.1.0.2.1.4.0-5 to NiFi version 1.2.0.3.0.0.0-453 of HDF 3.0 but I am having an issue when the node joins the cluster. I run a cluster with three NiFi nodes, I was trying a rolling upgrade to not stop all the nodes at the same time. I upgraded one of the nodes and tried to make it join the cluster. The node seemed to start up fine with no issues and said that the cluster had received the heartbeat. However, after the node joined the cluster the UI page in all three nodes started showing an error message saying "Unexpected error please check the logs for any errors". I went to the logs and I found the following warnings: 2017-06-14 11:36:49,279 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.s3.FetchS3Object is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,292 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.s3.PutS3Object is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,294 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.s3.DeleteS3Object is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,301 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.s3.ListS3 is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,310 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.sns.PutSNS is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,481 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.sqs.GetSQS is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,482 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.sqs.PutSQS is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,483 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.sqs.DeleteSQS is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,492 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.lambda.PutLambda is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,497 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.kinesis.firehose.PutKinesisFirehose is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,508 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.dynamodb.GetDynamoDB is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,510 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.dynamodb.PutDynamoDB is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,512 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.dynamodb.DeleteDynamoDB is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,517 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.kinesis.stream.PutKinesisStream is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,523 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.aws.cloudwatch.PutCloudWatchMetric is bundled with its referenced Controller Service APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:49,610 WARN [main] org.apache.nifi.nar.ExtensionManager Controller Service org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService is bundled with its supporting APIs org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService. The service APIs should not be bundled with the implementations.
2017-06-14 11:36:49,714 WARN [main] org.apache.nifi.nar.ExtensionManager Controller Service org.apache.nifi.jms.cf.JMSConnectionFactoryProvider is bundled with its supporting APIs org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition. The service APIs should not be bundled with the implementations.
2017-06-14 11:36:50,530 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.gcp.storage.PutGCSObject is bundled with its referenced Controller Service APIs org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:50,533 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.gcp.storage.FetchGCSObject is bundled with its referenced Controller Service APIs org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:50,534 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.gcp.storage.DeleteGCSObject is bundled with its referenced Controller Service APIs org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:50,535 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.gcp.storage.ListGCSBucket is bundled with its referenced Controller Service APIs org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:50,544 WARN [main] org.apache.nifi.nar.ExtensionManager Controller Service org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService is bundled with its supporting APIs org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService. The service APIs should not be bundled with the implementations.
2017-06-14 11:36:52,743 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.couchbase.GetCouchbaseKey is bundled with its referenced Controller Service APIs org.apache.nifi.couchbase.CouchbaseClusterControllerService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:52,746 WARN [main] org.apache.nifi.nar.ExtensionManager Component org.apache.nifi.processors.couchbase.PutCouchbaseKey is bundled with its referenced Controller Service APIs org.apache.nifi.couchbase.CouchbaseClusterControllerService. The service APIs should not be bundled with component implementations that reference it.
2017-06-14 11:36:52,748 WARN [main] org.apache.nifi.nar.ExtensionManager Controller Service org.apache.nifi.couchbase.CouchbaseClusterService is bundled with its supporting APIs org.apache.nifi.couchbase.CouchbaseClusterControllerService. The service APIs should not be bundled with the implementations.
However in the app-log file after those warnings everything else seemed to start up fine and the node was connected to the cluster, I was just not able to access the UI from any of the nodes. I saw no errors or warnings in the bootstrap or user logs. I wanted to see if there could be something wrong that I am doing to the cluster with the node configuration or if the problem is that I would have to upgrade all the nodes and start them together all in the same NiFi version 1.2.0.3.0.0.0-453 of HDF 3.0? Any insights on this would be incredible helpful.
... View more
Labels:
- Labels:
-
Apache NiFi
06-01-2017
11:28 PM
Hello, I am trying to pull an entire elastic search index using the ScrollElasticSearchHttp processor in NiFi and I configured the processor in the following way: However when I run the processor I get the following error: 18:22:47 EST
ERROR
31393db7-e718-1a79-ffff-ffffbf6ed7cd
ScrollElasticsearchHttp[id=31393db7-e718-1a79-ffff-ffffbf6ed7cd] ScrollElasticsearchHttp[id=31393db7-e718-1a79-ffff-ffffbf6ed7cd] failed to process session due to org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=9c712215-1871-46de-a0ca-3f14fc759a7a,claim=,offset=0,name=10963994969599232,size=0] is not the most recent version of this FlowFile within this session (StandardProcessSession[id=100101083029]): org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=9c712215-1871-46de-a0ca-3f14fc759a7a,claim=,offset=0,name=10963994969599232,size=0] is not the most recent version of this FlowFile within this session (StandardProcessSession[id=100101083029])
18:22:47 EST
WARNING
31393db7-e718-1a79-ffff-ffffbf6ed7cd
ScrollElasticsearchHttp[id=31393db7-e718-1a79-ffff-ffffbf6ed7cd] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
Can anyone help me understand what is causing the error? Do I need to up the connection time or change something in the configurations? Any help would be greatly appreciated!
... View more
Labels:
- Labels:
-
Apache NiFi
06-01-2017
03:43 PM
Thanks Matt, the QueryElasticsearchHTTP processor was very useful and helped me with the problem.
... View more
05-31-2017
04:45 PM
Hello, I am trying to build a flow that gets a file from elasticsearch, however since it is my first time using the FetchElasticSearch processor I am having a doubt about how to configure it. I don't understand what the Document Identifier means and what value am I suppose to give in this configuration. If someone could help in this I would greatly appreciate it.
... View more
Labels:
- Labels:
-
Apache NiFi
05-22-2017
08:19 PM
1 Kudo
I was able to figure it out. I used the EvaluateJsonPath processor and grabbed the 'Raw_Json' and the 'partition_date' column and then I used the AttributestoJson processor to turn those two attributes into a Json. Afterwards the Inferavroschema processor was able to infer the 'Raw_Json" column as a string and it is now putting it into the Hive table via HiveStreaming correctly.
... View more
05-22-2017
08:15 PM
@mqureshi I was sending "inferred.avro.schema" as an attribute and the input content was set to json
... View more
05-22-2017
12:50 AM
@Deepak B S , I did not find a NiFi fix for this error, however in my case I discovered that it was due to the server getting overloaded due to a high amount of messages and because of this some messages were getting backlogged and by the time they would reach the HandleHTTPRequest processor they had experied. I raised the number of concurrent tasks in the processor on NiFi side and used a load balancer distribution rule to balance the load that was getting into NiFi and that seemed to fix the issue for me. You can try to raise the number of concurrent tasks in your processor and on your flow if the error your getting is due to a load issue that can help.
... View more
05-19-2017
09:11 PM
1 Kudo
Hello, I am trying to create a flow in NiFi that takes a valid json file and puts it directly into a hive table using the PutHiveStreaming processor. My json looks something like the following: {
"Raw_Json": {
"SystemInfo": {
"Id": "a string ID",
"TM": null,
"CountID": "a string ID",
"Topic": null,
"AccountID": "some number",
"StationID": "some number",
"STime": "some Timestamp",
"ETime": "some Timestamp"
},
"Profile": {
"ID": "ID number",
"ProductID": "Some Number",
"City": "City Name",
"State": "State Name",
"Number": "XXX-XXX-XXXX",
"ExtNumber": null,
"Unit": null,
"Name": "Person Name",
"Service": "Purchase",
"AddrID": "00000000",
"Products": {
"Product": [{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
}]
}
},
"Total": {
"Amount": "some amount",
"Delivery": "some address",
"Estimate": "some amount",
"Tax": null,
"Delivery_Type": null
}
},
"partition_date":"2017-05-19"
}
I am getting the json, using the InferAvroSchema processor and from there converting the json to avro format by using the inferred avro schema and sending it into the PutHiveStreaming processor. My Flow looks something like this: The main goal is that I want all of the "Raw_Json" column to be dumped into one column in the hive table and the table will be partitioned by the "partition_date" column which will be the second column of the table. The problem is that for some reason NiFi is having problems inferring the nested json from the "Raw_Json" column and is dumping it like Null on the table as shown below: Does anyone know how could I make NiFi read the entire nested Json of the "Raw_Json" column as a string column and send it to the hive table? How could I create my own avro schema for it to do this? My main goal would be that the Raw_Json can be read as a string column. Any insight or ideas on how to fix this issue would be greatly appreciated!
... View more
Labels:
- Labels:
-
Apache NiFi
04-24-2017
08:50 PM
@Wynner, I was able to rejoin the cluster by moving the data in the repositories and restarting the node like you said. Thanks for the help!
... View more
04-24-2017
03:20 PM
@Wynner, all of the disk partitions are in less than 10% use and I just tested disconnecting and connecting the other nodes which use the exact same zookeeper connection string as the node that is giving problems and they rejoin the cluster with no issues or errors. Could something have caused a file to get corrupted on this node and make it be causing the error?
... View more
04-24-2017
02:58 PM
@Wynner, for this cluster I do not use embedded zookeeper as we have our own zookeeper cluster and that one is the one that is used to manage the NiFi cluster. However I checked zookeeper and everything seems fine on it. I will check the NiFi repositories and the space on the disk partitions.
... View more
04-24-2017
02:48 PM
Hello, I run a NiFi cluster with 3 nodes using NiFi-1.1.0 version. The cluster has been running with no issues for the last couple of months, however I checked it today and one of the nodes had suddenly disconnected and it won't join back to the cluster. I checked the logs of the node and the following error keeps appearing non stop in the logs: ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
2017-04-24 10:31:34,694 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
I had never encounter this issue before and wanted to know if someone could give me an idea of what could be causing it or how it could be fixed. I am a bit confused as no changes have been made to the configurations of the node or the cluster whatsoever and the other two nodes are working completely fine. Any insight on this issue would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
04-21-2017
04:12 PM
thanks a lot for the help! I will try this.
... View more