NiFi S2S extremely slow handshake

[#1]

I have a pretty simple setup.  We have two NiFi servers on different networks, and they need to transfer data between them.  I have site to site set up and working using HTTPS, but for some reason it seems like every single file transfer takes a minimum of 15 seconds.

A 1 byte file will take barely over 15000ms, at a transfer rate of a couple hundred bytes per second, or a much larger file will transfer in just under 30 seconds, at a transfer rate of about 85MB/s.  I'm guessing the actual transfer of the file is happening quickly, but the transfer speed takes into account the 15 second handshake....or whetever the process is that is taking 15 seconds each time.

I looked around and couldn't find anyone else experiencing quite the same thing.

I have tried using RAW instead of HTTPS, but I haven't found a working setup for that yet--every time I try I end up with the "Magic Header not found" message.

Here are some snippets of my config files, let me know if there is anything obviously wrong.  A suggestion for how exactly to set up a RAW connector would be appreciated as well, since I've heard those have better performance anyway.

Server A:

# Site to Site properties
nifi.remote.input.host=$localhostname (not a variable, my superiors are just paranoid about sharing too much information)
nifi.remote.input.secure=true
nifi.remote.input.socket.port=443
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

nifi.web.https.host=$localhost IP
nifi.web.https.port=8443
nifi.web.https.network.interface.default=
nifi.web.https.application.protocols=http/1.1
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=60 secs
nifi.web.request.ip.whitelist=$server b IP
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"

Server B:
# Site to Site properties
nifi.remote.input.host=
nifi.remote.input.secure=true
nifi.remote.input.socket.port=443
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

nifi.web.https.host=
nifi.web.https.port=8443
nifi.web.https.network.interface.default=
nifi.web.https.application.protocols=http/1.1
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=600 secs
nifi.web.request.ip.whitelist=$Server A IP
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"

[#2]

@tkchea 

NiFi Remote Process Groups (RPG) transfer FlowFiles and not just the FlowFile content. So depending on the amount of metatdata/attributes on the FlowFile. the amount transferred would be larger.

The RPG fetches Site-to-Site (S2S) details via a background thread the runs every 30 seconds regardless of existence of FlowFile.  These S2S details fetched will include details on the target NiFi (Number of nodes in target cluster, load on each node, RAW ports if configured, If HTTP is enabled, etc..).  These details are then used to facilitate the transfer of FlowFiles from client (RPG) and target NiFi (with Remote input or output ports).    

The actual transfer of FlowFile will either happen over the HTTPS port (used by a lot of other transactions) or via a RAW socket port depending on configuration.

Since a FlowFile consists of two parts (FlowFile Metadata and FlowFile Content), there is going to be disk and CPU I/O involved with writing to the flowfile_repository and content_repository.  So you may want to monitor those on both source and destination. 

When it comes to the mutual TLS handshake, NiFi is not doing anything special here.  The client certificate presented is used to identify the client and verify authorization to the send to or pull from a remote port.

You can also enable ssl handshake debug logging in the nifi bootstrap.conf file.

java.arg.ssldebug=-Djavax.net.debug=ssl,handshake

Of course you see all SSL handshakes including those when someone access the NiFi UI in the nifi-bootstrap.log file.   But this would allow you to see if you are seeing systematic slow TLS handshakes or only between these two networks.  You could also setup an RPG that sends to a remote input port on the same NiFi server.  The same TLS handshake will happen there as well.  Is it much faster (rules out an RPG issue.) 

If it ends up being the network between NiFi servers, you'll need to investigate there perhaps using something like wireshark may help.

Another test might involve using a postHTTP or InvokeHTTP to send to a ListenHTTP or HandleHTTPRequest processor on target server (can be setup to be secure or insecure using same keystore and truststore your NiFi's use).

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt