Attached are the logs that are related to my getKafka processor. Can someone please go through the same and let me know what might be going wrong!getkafka-processor-log.txt ... View more

I did the changes as specified in the above link(https://community.hortonworks.com/articles/28180/how-to-configure-hdf-12-to-send-to-and-get-data-fr.html) and configured getkafka processor. But I am getting below error when I start the getKafka processor. ********************Error Log********* 2017-08-24 10:10:19,964 INFO [test-consumer-group_storagev4-1c-1503583093158-c73a2251-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1503583093205] Added fetcher for partitions ArrayBuffer() 2017-08-24 10:10:20,169 WARN [test-consumer-group_storagev4-1c-1503583093158-c73a2251-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [test-consumer-group_storagev4-1c-1503583093158-c73a2251-leader-finder-thread], Failed to find leader for Set([indexing,0]) java.lang.NullPointerException: null ************************ ... View more

Nope. issue is not resolved! ... View more

We have that flow running fine with out Nifi in our current setup. We are trying to migrate to Nifi, hence build the flow in Nifi using Consume Kafka. ... View more

I tried get some help from blogs. But I am unable to find the relevant information to resolve the issue. It would be helpful if any one have configured Consumer Kafka processor in Nifi successfully to process thousands of records per minutes can share the way the processor is ocnfigured! Thanks, Anil. ... View more

@Pierre Leroy when you are referring to download content from the URL, in general GET method is supported to get content from the webservice. I am not sure if you are configuring the POST request wrongly. "HTTP error 405 method not allowed" means URL exists with webservice but you are not authorized to access/perform that action. Please check if you need to specify any authorization headers to perform the action. I would suggest to get more details about the URL like uri, http method, is it ssl encrypted, any authorization fields required to access the URL etc and configure the Nifi processor to fulfill the same! ... View more

Your are right. I am just accepting the HTTP request and sending the response. Queue is not filling up, it is HandleHttpRequest not processing the requests fast enough. I was under an impression that HandleHttpRequest should be able to process upto the performance of Jetty server . But unable to see such performance. Yes, I do have enough resources to process 100 concurrent tasks. I haven't increased overall timer driven thread pool. I am new to this settings. I try to get context about these settings timer driven and event driven. Yes, I am using load balancer Nginx to request across nodes in the Nifi cluster. I am not sure if I get your last question correctly. I am just sending status 200 as Http response. My bench mark sample results with 1000 concurrent results and total count of 10000 records are: Load on standalone single: took about two minutes, no http timeouts Load on all the node individually (i.e 10000 * 3 nodes): took about 45 mins. No http timeouts Load on Nginx load balancer to use all Nifi nodes (just 1000 concurrent and 10000 records): took about 5 mins to process. I see timeout errors(504 status) as well from with Nginx. I feel these results does not makes sense as apache and jetty kind of servers can process thousands of records in seconds. Please let me know your views. Thank you for your prompt response and sharing your knowledge. Awaiting for your reply. ... View more

Matt Clarke thanks for the latest response. Its informative and gives more understanding about regular expressions! ... View more