Member since
03-08-2017
17
Posts
0
Kudos Received
0
Solutions
08-01-2018
08:31 AM
I could solve this issue by making the execution for handlehttprequest to run only on primary node instead of all nodes. Post that I could see that number of threads are not crossing permissible limit and the flow and Nifi services are running fine with out any issues. Not sure, but I think handlehttprequest can't run on a cluster mode with multiple nodes as the endpoint url:port specified will be unique and some TCP sockets are getting locked state and increased the nproc value more than specified value for the nifi user. Thanks, Vish
... View more
07-31-2018
01:58 PM
Hi Team, I have Handlehttprequest processor pointing to one of the host where nifi is running. I have Nifi cluster running on 3 nodes on HDP. When ever I start the Process group containing Handlehttprequest and some set of transformation followed by Handlehttpresponse, with in few minutes, Nifi UI becoming unresponsive and I get the below error message : javax.ws.rs.ProcessingException: java.net.SocketException: Unexpected end of file from server I have followed the Nifi set up guide to increase the ulimit for nproc nofile for the nifi user . But still I feel , nproc limit is crossing the limits. Also looking at the nifi-app.log , I can see below error message on one of the node. 2018-07-31 17:13:40,298 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=74ee1128-2de6-3979-a818-2b598186f7aa] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server: {} org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server at org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:488) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.eclipse.jetty.util.MultiException: Multiple exceptions at org.eclipse.jetty.server.Server.doStart(Server.java:416) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.processors.standard.HandleHttpRequest.initializeServer(HandleHttpRequest.java:425) at org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:484) ... 12 common frames omitted Suppressed: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:298) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.server.Server.doStart(Server.java:431) ... 15 common frames omitted Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:476) at org.eclipse.jetty.util.thread.QueuedThreadPool.doStart(QueuedThreadPool.java:116) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:452) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:419) ... 15 common frames omitted 2018-07-31 17:13:41,297 INFO [Timer-Driven Process Thread-4] org.eclipse.jetty.server.Server jetty-9.4.3.v20170317 2018-07-31 17:13:41,297 ERROR [Timer-Driven Process Thread-4] o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=74ee1128-2de6-3979-a818-2b598186f7aa] HandleHttpRequest[id=74ee1128-2de6-3979-a818-2b598186f7aa] failed to process due to org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server; rolling back session: {} org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server at org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:488) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.eclipse.jetty.util.MultiException: Multiple exceptions at org.eclipse.jetty.server.Server.doStart(Server.java:416) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.processors.standard.HandleHttpRequest.initializeServer(HandleHttpRequest.java:425) at org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:484) ... 12 common frames omitted Suppressed: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:298) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.server.Server.doStart(Server.java:431) After few minutes, I can not even stop the nifi services from ambari and I have to manually kill the nifi process and then restart the nifi services from ambari. Is there any issues with Handlehttprequest while running it on Nifi cluster mode? Is there any thread/process that is getting built and not able to close based on any time out? Though I see issues with Nifi UI, I can still make curl post request to endpoint url specified in Handlehttprequest and I do get the response back from Handlehttpresponse. I have increased the default timeout properties and jvm memory parameters to optimal level. Any idea on the issue that I am facing and any help in providing the direction to fix the issue is greatly appreciated. Thanks, Vish
... View more
Labels:
- Labels:
-
Apache NiFi
07-27-2018
01:40 PM
sorry for my ignorance. I could resolve this issue by removing one additional space in " list" while wring the flowfile content generated by groovy script. Now I am able to fetch the data from invoke http by passing flowfile content from executescript. Thanks for the support. Regards, Vish
... View more
07-27-2018
12:34 PM
In addition to my earlier query, I could call druid endpoint to fetch the data based on the flow file content holding the json record. flowfilecontent-->>replacetext-->updateattribute-->>Invokehttp-->putfile in replace text I gave replacement value as below: {"queryType":"scan","dataSource":"nifi_kafka_druid_test","resultFormat":"list","columns":["station_id","mac_address","result","test_station_name"],"intervals":["2018-01-01/2018-02-09"],"filter":{"type":"selector","dimension":"sn","value":"GGJ807405ZBJLR6AB"}} and added the attribute mime.type=application/json in update attribute I can call the invokehttp to druid end point to fetch the data by endpoint url: [{"segmentId":"nifi_kafka_druid_test_2018-02-07T00:00:00.000Z_2018-02-07T00:05:00.000Z_2018-07-17T13:27:30.419Z","columns":["station_id","mac_address","result","test_station_name"],"events":[{"station_id":"LMMP_IQE-1FT-01_42_AE-1","mac_address":"ABC","result":"PASS","test_station_name":"EPI_DIE"}]}] But when I try to pass flow file content of the groovy script to invokehttp I am not getting any response. I also changed the groovy script to write the content of the json in single line instead of multiline pretty format. Looking through the data provenance of the sample pipeline mentioned above to the actual pipeline with the execute script, I do not see any difference in the flowfile content passed to invokehttp. Not sure, what am I doing wrong. Please suggest. Thanks, Vish
... View more
07-27-2018
10:22 AM
Thanks @Matt Burgess for the help. I could use the script that you shared. Number of parameters are dynamic in nature. Now I could see the content in output as below: {
"queryType": "scan",
"dataSource": "nifi_kafka_druid_test",
"resultFormat": " list",
"columns": [
"test_station_name",
"station_id",
"result",
"mac_addresss"
],
"intervals": [
"2018-01-01/2018-02-09"
],
"filter": {
"type": "selector",
"dimension": "sn",
"value": "GH6747246T4JLR6AZ"
}
} Now I am trying to pass this to invokehttp to my druid broker endpoint url. But I am not able to get the response back from invokehttp. Here if I use curl command: curl -X POST 'brokerip:port/druid/v2/?pretty' -H 'Content-Type:application/json' -d'{ "queryType": "scan", "dataSource": "nifi_kafka_druid_test", "resultFormat": "list", "columns":["station_id","mac_address","result","test_station_name"], "intervals": [ "2018-01-01/2018-02-09" ],"filter": { "type": "selector", "dimension": "sn", "value":"GGJ807405ZBJLR6AB" } }' I get the response:
[ {
"segmentId" : "nifi_kafka_druid_test_2018-02-07T00:00:00.000Z_2018-02-07T00:05:00.000Z_2018-07-17T13:27:30.419Z",
"columns" : [ "station_id", "mac_address", "result", "test_station_name" ],
"events" : [ {
"station_id" : "LMMP_IQE-1FT-01_42_AE-1",
"mac_address" : "ABC",
"result" : "PASS",
"test_station_name" : "EPI_DIE"
} ] I want to perform the same based on the flowfile content produced by executescript. Am I missing some thing? how should I pass the data content to invokehttp processor? Any help is appreciated. Thanks, Vish
... View more
07-27-2018
04:51 AM
Thanks a lot @Matt Burgess for the details on the current limitations and the Jira. I would look at the solution provided by you on the other thread. Appreciate your help on the same. Regards, Vish
... View more
07-26-2018
08:46 AM
Hi All, I am using combination of Handlehttprequest and Handlehttpresponse processors to get the rest end point to the client application. I have issues with the below type of curl post request. Handlehttprequest is not creating the array of parameters and is just keeping last value of the parameter values. eg: curl -L -X POST "http://url:port/data?sn=GH6747246T4JLR6AZ&c=QUERY_RECORD&p=test_station_name&p=station_id&p=result&p=mac_addresss" When I look at the data provenance I see that under attribute values : I see http.param.p holding only the first value(test_station_name) and http.query.param.p holding only the last value (mac_address) Is there way to make this hold array of all the values for the parameter p? Other way I thought is of using http.query.string which has all the parameter passed and use JoltJSONTRansform to transform the required inputs for me. But JoltJSONTransform spec for achieving this seems to be little tricky. ANy help is greatly appreciated. Thanks, Vish
... View more
Labels:
- Labels:
-
Apache NiFi
07-26-2018
06:22 AM
Hi All, Need help in JoltJSONTransform spec that can be used to convert the input to output. I have tried to use map to List and other syntax, but was not been successful so far. Any help is appreciated. Input: {"params":"sn=GH6747246T4JLR6AZ&c=QUERY_RECORD&p=test_station_name&p=station_id&p=result&p=mac_addresss"} Output: { "queryType": "scan", "dataSource": "xyz", "resultFormat": "list", "columns":["test_station_name","station_id","result","mac_address"], "intervals": [ "2018-01-01/2018-02-09" ],"filter": { "type": "selector", "dimension": "sn", "value":"GH6747246T4JLR6AZ" } } Except for the content inside Columns and dimension and value attributes rest of the fields are hardcoded. Please suggest. Thanks, Vish
... View more
Labels:
- Labels:
-
Apache NiFi