Member since
09-29-2016
54
Posts
9
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1755 | 07-03-2017 09:12 PM | |
1403 | 05-22-2017 08:19 PM |
10-30-2017
06:36 PM
@Xu Zhe, yes the cluster is fully stable now.
... View more
10-27-2017
11:47 AM
Hi @Xu Zhe As mentioned in the comments by Wynner, the solution I used was to clean up the NiFi repositories and restart the cluster nodes at the same time.
... View more
09-06-2017
03:38 PM
Hi @Matt Burgess, I was able to connect successfully following your guidelines. I had to call the elasticsearch folder from the X-Pack Zip for the ES version 5.0.1 along with the 5.0.1 Transport Jar in the X-Pack Transport Location property as a comma-delimited list. Thanks for all the help, greatly appreciate it!
... View more
08-28-2017
07:16 PM
1 Kudo
Hello, I am trying to send feed from nifi version nifi-1.2.0.3.0.0.0-453 to elasticsearch version 5.5.1 however I keep getting the following error: PutElasticsearch5[id=f4da516d-2b44-1955-0000-00000a063fa3]
PutElasticsearch5[id=f4da516d-2b44-1955-0000-00000a063fa3] failed to process due
to org.apache.nifi.processor.exception.ProcessException:
org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be
loaded and/or configured; rolling back session:
org.apache.nifi.processor.exception.ProcessException:
org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be
loaded and/or configured
The full error log is the following: ERROR [Timer-Driven Process Thread-16] o.a.n.p.elasticsearch.PutElasticsearch5 PutElasticsearch5[id=f4da516d-2b44-1955-0000-00000a063fa3] Failed to create Elasticsearch client due to org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be loaded and/or configured: org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be loaded and/or configured
org.apache.nifi.processor.exception.ProcessException: X-Pack plugin could not be loaded and/or configured
at org.apache.nifi.processors.elasticsearch.AbstractElasticsearch5TransportClientProcessor.getTransportClient(AbstractElasticsearch5TransportClientProcessor.java:222)
at org.apache.nifi.processors.elasticsearch.AbstractElasticsearch5TransportClientProcessor.createElasticsearchClient(AbstractElasticsearch5TransportClientProcessor.java:170)
at org.apache.nifi.processors.elasticsearch.AbstractElasticsearch5Processor.setup(AbstractElasticsearch5Processor.java:94)
at org.apache.nifi.processors.elasticsearch.PutElasticsearch5.onTrigger(PutElasticsearch5.java:175)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.nifi.processors.elasticsearch.AbstractElasticsearch5TransportClientProcessor.getTransportClient(AbstractElasticsearch5TransportClientProcessor.java:215)
... 15 common frames omitted
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.xpack.XPackPlugin
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:451)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:116)
at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:81)
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:106)
at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:228)
at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:69)
at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.<init>(PreBuiltXPackTransportClient.java:50)
at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.<init>(PreBuiltXPackTransportClient.java:46)
... 20 common frames omitted
I tried following the suggestions in the link: https://community.hortonworks.com/questions/81606/where-should-i-store-files-used-by-processors-in-a.html , but I am unable to fix the issue. I tried putting the x-pack jar and the transport jar for the elasticsearch version 5.5.1 on the X-Pack Transport Location configuration of the PutElasticsearch5 processor and I also tried using the x-pack jar and the transport jar of the elasticsearch version 5.0.1, however nothing seems to be working. Any suggestions on the issue would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
08-07-2017
08:54 PM
Hello, I am receiving around 300k-400k messages per day in NiFi version Nifi-1.2.0.3.0.0.0-453. The messages are coming as a Json and the goal is to put them into a hive table in near real time. I have built the following flow which is attached as an image in this post. The flow is working fine and doing everything like it is supposed to. The issue is that is not writing to putHiveStreaming fast enough and as a result messages start getting queued up in the flow. I have been playing around with the Transaction per Batch and Records per Transaction configuration properties from the putHiveStreaming but I seem to be unable to get the putHiveStreaming processor to write faster but it does not seem to be working and we are getting messages a lot faster than what we are writing. Is there a way to configure the putHiveStreaming processor so that it can handle this type of load of 300k-400k messages per day at around 3 to 4 messages per second fast enough? Any insight on this issue will be deeply appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
07-06-2017
08:12 PM
@Matt Clarke that seemed to fix the issue, thanks a lot!
... View more
07-05-2017
10:22 PM
Full error trace: 2017-07-05 17:04:12,336 WARN [Replicate Request Thread-8] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/process-groups/04a08abd-d360-3539-ad01-835bc54126b0 due to {}
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:611)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:822)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 12 common frames omitted
... View more
07-05-2017
09:42 PM
Hello, I am on the process of changing the nodes of our current nifi cluster for completely nifi dedicated nodes. I usually ran a 4 node nifi cluster and today I added 4 new nodes to that cluster. The main goal is to take out the old 4 nodes and to keep the new 4 nifi dedicated nodes in the cluster. All the nodes in the cluster are currently running on the nifi version nifi-1.2.0.3.0.0.0-453. I added the 4 new nodes successfully and gave them the proxy permissions and everything. I am able to log in to the new nodes and access the UI from them and they connected to the cluster without any issues. However whenever I try to access a specific process group from one of the 4 new nodes I keep getting the following error: This is only happening whenever I try to access that specific process group from any of the new nodes. If I access it from any of the old nodes no error happens and I can access the process group just fine. I looked at the logs from the new nodes and this is what I see: 2017-07-05 17:10:56,745 WARN [Replicate Request Thread-10] o.a.n.c.c.h.r.ThreadPoolRequestReplicator
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:611)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:822)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 12 common frames omitted
Does anyone have any ideas of what could be causing this? Is there a configuration that I have to change to avoid this error? or a permission that I have to give these nodes? Any insights would be extremely helpful.
... View more
Labels:
- Labels:
-
Apache NiFi
07-03-2017
09:12 PM
I was able to resolve the problem. The issue was that my kbr5.conf file that I called in the nifi.kerberos.krb5.file line in the nifi.properties file, was not configured properly. After properly configuring it and calling it in the nifi.properties file I was able to successfully send to kafka. I also had to use the PublishKafka_0_10 processor instead of the PutKafka processor since the version of kafka I was trying to connect to was 0.10.
... View more
07-03-2017
06:14 PM
@Shashank Chandhok
I did some changes to the jass.conf file but I am still getting the same error. This is the new jass.conf file:
... View more