Support Questions
Find answers, ask questions, and share your expertise

[NIFI] Node disconnected from cluster : NIFI-API

Explorer

hi all,

do you know why all nodes disconnected from cluster when they've trying an curl nifi-api/data-provenance ?

here my python code, it execute all 5 minutes on all nodes of clusters:

    headers = {"Content-Type": "application/json", "Accept": "application/json, text/javascript"}
    conn=httplib.HTTPSConnection('nifi002', 9443, key_file=None, cert_file="cert_file")
    conn.request("POST", "/nifi-api/provenance", json.dumps(params), headers)
    response = conn.getresponse()
    result = json.loads(response.read())
    conn.close()

    conn=httplib.HTTPSConnection('nifi002', 9443, key_file=None, cert_file="cert_file")
    conn.request("GET", "/nifi-api/provenance/" + result['provenance']['id'])
    res = conn.getresponse()
    result = json.loads(res.read())
    conn.close()

    conn=httplib.HTTPSConnection('nifi002', 9443, key_file=None, cert_file="cert_file")
    conn.request("DELETE", "/nifi-api/provenance/" + result['provenance']['id'])
    res = conn.getresponse()
    conn.close()

here errors:

2017-05-23 14:34:57,021 INFO [Clustering Tasks Thread-3] o.a.n.c.c.ClusterProtocolHeartbeater After receiving heartbeat response, updated status of nifi002.current.rec.mapreduce.m0.p.fti.net:9443 to NodeConnectionStatus[nodeId=nifi002.current.rec.mapreduce.m0.p.fti.net:9443, state=DISCONNECTED, Disconnect Code=Failed to Service Request, Disconnect Reason=Failed to process request POST /nifi-api/provenance, updateId=355]
2017-05-23 14:34:57,022 INFO [Clustering Tasks Thread-3] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-05-23 14:34:56,801 and sent to nifi003:11443 at 2017-05-23 14:34:57,022; send took 220 millis
2017-05-23 14:34:59,383 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 9bac3908-0eeb-4236-bf43-dbe18962ebb3 (type=DISCONNECTION_REQUEST, length=649 bytes) from nifi002.current.rec.mapreduce.m0.p.fti.net:9443 in 77 millis
2017-05-23 14:34:59,385 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Received disconnection request message from manager with explanation: Failed to process request POST /nifi-api/provenance
2017-05-23 14:34:59,385 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Disconnecting node.

2017-05-23 14:34:55,138 INFO [Process Cluster Protocol Request-4] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 847146bf-b6eb-47d5-942e-25e90723713d (type=DISCONNECTION_REQUEST, length=649 bytes) from nifi002:9443 in 1746 millis
2017-05-23 14:34:55,138 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Received disconnection request message from manager with explanation: Failed to process request POST /nifi-api/provenance
2017-05-23 14:34:55,138 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Disconnecting node.
2017-05-23 14:34:56,819 INFO [Disconnect from Cluster] o.apache.nifi.controller.FlowController Cluster State changed from Clustered to Not Clustered
2017-05-23 14:34:56,835 INFO [Process Cluster Protocol Request-6] o.a.n.c.p.impl.SocketProtocolListener Finished processing request ca1d66af-51d6-44c0-b840-9ee9396ff451 (type=DISCONNECTION_REQUEST, length=649 bytes) from nifi002:9443 in 1754 millis
2017-05-23 14:34:56,836 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Received disconnection request message from manager with explanation: Failed to process request POST /nifi-api/provenance
2017-05-23 14:34:56,839 INFO [Disconnect from Cluster] o.a.n.c.l.e.CuratorLeaderElectionManager This node is no longer registered to be elected as the Leader for Role 'Primary Node'
2017-05-23 14:34:56,844 INFO [Disconnect from Cluster] o.a.n.c.l.e.CuratorLeaderElectionManager This node is no longer registered to be elected as the Leader for Role 'Cluster Coordinator'
2017-05-23 14:34:56,845 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Node disconnected.
2017-05-23 14:34:56,845 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Disconnecting node.
2017-05-23 14:34:56,872 INFO [Disconnect from Cluster] o.a.n.c.l.e.CuratorLeaderElectionManager Cannot unregister Leader Election Role 'Primary Node' becuase that role is not registered
2017-05-23 14:34:56,872 INFO [Disconnect from Cluster] o.a.n.c.l.e.CuratorLeaderElectionManager Cannot unregister Leader Election Role 'Cluster Coordinator' becuase that role is not registered
2017-05-23 14:34:56,872 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Node disconnected.
2017-05-23 14:34:57,021 INFO [Clustering Tasks Thread-3] o.a.n.c.c.ClusterProtocolHeartbeater After receiving heartbeat response, updated status of nifi002:9443 to NodeConnectionStatus[nodeId=nifi002:9443, state=DISCONNECTED, Disconnect Code=Failed to Service Request, Disconnect Reason=Failed to process request POST /nifi-api/provenance, updateId=355]
2017-05-23 14:34:57,022 INFO [Clustering Tasks Thread-3] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-05-23 14:34:56,801 and sent to nifi003:11443 at 2017-05-23 14:34:57,022; send took 220 millis
2017-05-23 14:34:59,383 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 9bac3908-0eeb-4236-bf43-dbe18962ebb3 (type=DISCONNECTION_REQUEST, length=649 bytes) from nifi002:9443 in 77 millis
2017-05-23 14:34:59,385 INFO [Disconnect from Cluster] o.a.nifi.controller.StandardFlowService Received disconnection request message from manager with explanation: Failed to process request POST /nifi-api/provenance

thanks for helps.

3 REPLIES 3

Contributor

Hi @mayki wogno

I would suggest performing the provenance query through the NiFi UI and looking at the network tab to see the exact requests, payloads, headers, etc that the web application uses. There may be a discrepancy between that and the python requests you're making.

Explorer

@brosander

I'm already use NIFI UI for making my python script, i don't see the different with curl via NIFI UI

curl "https://nifi002:9443/nifi-api/provenance" -H "Host: nifi002:9443" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0" -H "Accept: application/json, text/javascript, */*; q=0.01" -H "Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3" --compressed -H "DNT: 1" -H "Content-Type: application/json" -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJtaHV5bmhARlRJLk5FVCIsImlzcyI6IktlcmJlcm9zUHJvdmlkZXIiLCJhdWQiOiJLZXJiZXJvc1Byb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoibWh1eW5oQEZUSS5ORVQiLCJraWQiOjEsImV4cCI6MTQ5NTY1MzAxMCwiaWF0IjoxNDk1NjA5ODEwfQ.6YjxqeNSDto8yOgJkxBNNbufXi3-XlOSV2dZP5tj2Tc" -H "X-Requested-With: XMLHttpRequest" -H "Referer: https://nifi002:9443/nifi/provenance?componentId=4e043ebd-8c0e-1024-84e6-e1f8b495d088" -H "Content-Length: 115" -H "Proxy-Authorization: Basic bWh1eW5oOm0xZkV2cm5oIQ==" -H "X-ClickOnceSupport: ( .NET CLR 3.5.30729; .NET4.0E)" -H "Connection: keep-alive"

{"provenance":{"request":{"maxResults":1000,"searchTerms":{"ProcessorID":"4e043ebd-8c0e-1024-84e6-e1f8b495d088"}}}}
    
POST /nifi-api/provenance HTTP/1.1
Host: nifi002:9443
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Content-Type: application/json
Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJtaHV5bmhARlRJLk5FVCIsImlzcyI6IktlcmJlcm9zUHJvdmlkZXIiLCJhdWQiOiJLZXJiZXJvc1Byb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoibWh1eW5oQEZUSS5ORVQiLCJraWQiOjEsImV4cCI6MTQ5NTY1MzAxMCwiaWF0IjoxNDk1NjA5ODEwfQ.6YjxqeNSDto8yOgJkxBNNbufXi3-XlOSV2dZP5tj2Tc
X-Requested-With: XMLHttpRequest
Referer: https://nifi002:9443/nifi/provenance?componentId=4e043ebd-8c0e-1024-84e6-e1f8b495d088
Content-Length: 115
X-ClickOnceSupport: ( .NET CLR 3.5.30729; .NET4.0E)
Connection: keep-alive    
    

Contributor

@mayki wogno

Do you see the same behavior when performing the operation through the UI?