Support Questions
Find answers, ask questions, and share your expertise

a node in cluster get disconnected when user try to modify the flow

a node in cluster get disconnected when user try to modify the flow

New Contributor

When I try to remove a funnel from the data flow, I got this error which is fine

Node nifi003:9443 is unable to fulfill this request due to: Funnel d3a13ca2-c222-1304-ffff-ffff9af6cba9 is the destination of another component

The problem is one of nifi node get disconnected, here is the log in nifi-app.log

2016-09-14 18:13:39,853 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Received disconnection request message from manager with explanation: Failed to process request DELETE /nifi-api/funnels/d3a13ca2-c222-1304-ffff-ffff9af6cba9 2016-09-14 18:13:39,853 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Disconnecting node. 2016-09-14 18:13:39,855 INFO [Disconnect from Cluster] o.apache.nifi.controller.FlowController Cluster State changed from Clustered to Not Clustered

I already set permission for all nifi nodes to view/modify the component, but even if I missed something when setup permission, how is this possible that some user interaction in UI can break my cluster?

Update 1:

The logs on nifi003

2016-09-15 19:20:08,644 WARN [Replicate Request Thread-1] o.a.n.c.c.node.NodeClusterCoordinator All nodes failed to process URI DELETE /nifi-api/process

ors/896636ca-f911-18ec-8bdc-8152edd29bef. As a result, no node will be disconnected from cluster

2016-09-15 19:20:08,644 DEBUG [Replicate Request Thread-6] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Replicating request to DELETE https://nifi003:9443/

nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef, request ID = null, headers = {Origin=https://52.36.34.152, Cookie=BCSI-CS-88ceaaeaec1fb52f=2,

Accept=application/json, text/javascript, */*; q=0.01, X-Requested-With=XMLHttpRequest, X-Cluster-Id-Generation-Seed=2f4ab8b6-0157-1000-edce-2de38dee995

a, User-Agent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2552.0 Safari/537.36, Referer=https://52.36.34

.152/nifi/, Connection=close, Host=52.36.34.152, Accept-Encoding=gzip, deflate, sdch, X-Request-Replicated=true, Authorization=Bearer eyJhbGciOiJIUzI1Ni

J9.eyJzdWIiOiJkbmd1eWVuIiwiaXNzIjoiV1FMb2dpbklkZW50aXR5UHJvdmlkZXIiLCJhdWQiOiJXUUxvZ2luSWRlbnRpdHlQcm92aWRlciIsInByZWZlcnJlZF91c2VybmFtZSI6ImRuZ3V5ZW4iL

CJraWQiOjEsImV4cCI6MTQ3Mzk5ODYyNSwiaWF0IjoxNDczOTU1NDI1fQ.hJv_WCNhKvG3KEiLNg5E7t1YMCmCHJnRJSqV3x4pfRI, X-Cancel-Transaction=true, X-ProxiedEntitiesChain

=<dnguyen><CN=nifi002, OU=NIFI>, X-RequestTransactionId=1c03ec27-fbfc-4278-9188-cb8d0d0b771c, Accept-Language=en-US,en;q=0.8, X-ProxyScheme=https, Conte

nt-Type=application/x-www-form-urlencoded}

2016-09-15 19:20:08,644 DEBUG [Replicate Request Thread-1] o.a.n.c.c.h.r.StandardAsyncClusterResponse Received response 1 out of 3 for 1c03ec27-fbfc-427

8-9188-cb8d0d0b771c from nifi001:9443

2016-09-15 19:20:08,644 DEBUG [Replicate Request Thread-1] o.a.n.c.c.h.r.StandardAsyncClusterResponse Received response 2 out of 3 for 1c03ec27-fbfc-4278-9188-cb8d0d0b771c from nifi003:9443

2016-09-15 19:20:08,644 DEBUG [Replicate Request Thread-1] o.a.n.c.c.h.r.StandardAsyncClusterResponse Received response 3 out of 3 for 1c03ec27-fbfc-4278-9188-cb8d0d0b771c from nifi002:9443

2016-09-15 19:20:08,644 DEBUG [Replicate Request Thread-1] o.a.n.c.c.h.r.StandardAsyncClusterResponse Notifying all that merged response is ready for 1c03ec27-fbfc-4278-9188-cb8d0d0b771c

2016-09-15 19:20:08,646 WARN [Replicate Request Thread-1] o.a.n.c.c.node.NodeClusterCoordinator The following nodes failed to process URI /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef '[nifi001:9443]'. Requesting each node disconnect from cluster.

2016-09-15 19:20:08,646 INFO [Replicate Request Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Requesting that nifi001:9443 disconnect due to Failed to process request DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef

2016-09-15 19:20:08,646 INFO [Replicate Request Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi001:9443 changed from NodeConnectionStatus[nodeId=nifi001:9443, state=CONNECTED, updateId=81] to NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Failed to Service Request, Disconnect Reason=Failed to process request DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef, updateId=82]

2016-09-15 19:20:08,646 DEBUG [Replicate Request Thread-1] o.a.n.c.c.node.NodeClusterCoordinator State of cluster nodes is now {nifi001:9443=NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Failed to Service Request, Disconnect Reason=Failed to process request DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef, updateId=82], nifi002:9443=NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTED, updateId=63], nifi003:9443=NodeConnectionStatus[nodeId=nifi003:9443, state=CONNECTED, updateId=14]}

2016-09-15 19:20:08,646 DEBUG [Replicate Request Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Notifying all nodes that status changed from NodeConnectionStatus[nodeId=nifi001:9443, state=CONNECTED, updateId=81] to NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Failed to Service Request, Disconnect Reason=Failed to process request DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef, updateId=82]

2016-09-15 19:20:08,647 WARN [NiFi Web Server-17407] c.s.j.spi.container.servlet.WebComponent A servlet request, to the URI https://52.36.34.152/nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef?version=4&clientId=2e9..., contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

2016-09-15 19:20:08,734 DEBUG [Clustering Tasks Thread-1] org.apache.nifi.cluster.heartbeat Generated heartbeat

2016-09-15 19:20:08,736 DEBUG [Process Cluster Protocol Request-8] o.a.n.c.p.impl.SocketProtocolListener Received request dcb9996e-3556-468c-b59c-5a641378dc7d from *******

2016-09-15 19:20:08,739 DEBUG [Notify Cluster of Node Status Change-1] o.a.n.c.p.i.StandardClusterCoordinationProtocolSender Notified nifi002:9443 of status change NodeStatusChangeMessage[nodeId=nifi001:9443, status=NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Failed to Service Request, Disconnect Reason=Failed to process request DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef, updateId=82]]

2016-09-15 19:20:08,740 DEBUG [Replicate Request Thread-1] o.a.n.c.c.h.r.ThreadPoolRequestReplicator For DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef (Request ID 1c03ec27-fbfc-4278-9188-cb8d0d0b771c), minimum response time = 0, max = 8, average = 4.666666666666667 ms

2016-09-15 19:20:08,740 DEBUG [Replicate Request Thread-1] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef (Request ID 1c03ec27-fbfc-4278-9188-cb8d0d0b771c):

nifi001:9443: 0 millis

nifi002:9443: 8 millis

nifi003:9443: 6 millis

Calculated diff between current cluster status and node cluster status as follows: Node: [NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Failed to Service Request, Disconnect Reason=Failed to process request DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef, updateId=83], NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTED, updateId=63], NodeConnectionStatus[nodeId=nifi003:9443, state=CONNECTED, updateId=14]] Self: [NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Failed to Service Request, Disconnect Reason=Failed to process request DELETE /nifi-api/processors/896636ca-f911-18ec-8bdc-8152edd29bef, updateId=83], NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTED, updateId=63], NodeConnectionStatus[nodeId=nifi003:9443, state=CONNECTED, updateId=14]] Difference: []

5 REPLIES 5

Re: a node in cluster get disconnected when user try to modify the flow

Can you find any errors in the nifi-app.log on nifi003? We need to figure out why that node disconnected from the cluster.

Re: a node in cluster get disconnected when user try to modify the flow

New Contributor

Yes I've added logs from nifi003. Do you need to increase log level of any modules?

Re: a node in cluster get disconnected when user try to modify the flow

Also, are you running nifi-1.0.0 or nifi-1.0.0-beta?

Re: a node in cluster get disconnected when user try to modify the flow

New Contributor

I'm using nifi-1.0.0

Re: a node in cluster get disconnected when user try to modify the flow

We are getting the same problem. Found any solution or root cause to this issue?