Sending data from local pc to remote process group on azure

[#1]

Hi folks,

I cannot send data from my local pc to a remote process group on an azure vm (Ubuntu 18.04) and back.
Both nifi instances are unsecured. I try to send with http. The local nifi-rpg sees the remote input and output ports and it is able to visualize their current states (Running, Stopped). However I am unable to send data from or to the rpg.

I assume the fault lies withing the localhost property of the vm.
Setting it to the public ip or domain name will result in a crash of nifi. Setting it to the internal ip does not change the behaviour of the rpg.
Sadly I cannot ping the public IP address from within the VM and I do not see a way for port forwarding.

Does someone know how to enable remote process groups on an azure linux vm? It seems more like an azure than a nifi issue, but the lack of messages I find online tells me that I am missing something simple.

Should it usually work with setting the localhost to the public (or strangely private) ip?
Should an azure vm usually be able to ping it's own public ip?
Is it common for nifi to fail in startup if the localhost property is set to an unreachable ip?

Greetings

Frank

[#2 · Accepted solution]

@Frank Gaibler

-

The RPG works in the following way:

1. User configures RPG with target NiFi URL (Optional: if target is a NiFi cluster a comma separated list of each node's URL can be provided). User decides if HTTP or RAW transport protocols will be used.

2. Upon clicking "ADD", the RPG will try connecting to the target NiFi in order to retrieve Site-to-Site (S2S) details. These details include things like the number of nodes, node connection info, available remote input/output ports, and current load on each node (if target is cluster)

---- Up to this point it sounds like you are working ----

The node connection information is what matter to you here and it comes from the following properties.

The properties you need to be most concerned with are:

1. nifi.remote.input.host= <-- here you can explicitly define the hostname you want sent back to any client requesting a S2S connection. It is this hostname that will be used by the configured transport protocol. leaving this blank will result in Java trying to determine the hostname and in many case this can lead to an internal hostname or even localhost being used.

2. nifi.remote.input.socket.port= <-- If RAW transport protocol is selected, this is the port all FlowFile transmission will occur over. Make sure it is open through any firewalls.

-

Thanks,

Matt

-

When an "Answer" addresses/solves your question, please select "Accept" beneath that answer. This encourages user participation in this forum.

[#3]

@Frank Gaibler

-

Forum tip: Avoid starting a new "Answer" to respond to an existing answer. Instead click "Add comment" or "Reply". This makes following a thread/line of thought easier. There is also no guarantee to order of individual answers which can make following the thread difficult.

-

You are correct that the "nifi.remote.input.host=" properties should be configured with the public IP or public hostname for the server. When you configured your node with a public IP or hostname, What error are you seeing when you say NiFi crashes shortly after restart when the above property is configured?

-

Thanks,

Matt

[#4]

@Matt Clarke

After going through logfiles, I assume I have one of the right parts with the failure:

It did not produce any output with "nifi.sh run". I was just thrown out of the process and back to console.

However I do not know what I did differently from all the other times. It now worked without any issue.

Everything works for now. I will post updates if I am able to reproduce the issue again. I really have no clue what went different from before.

[#5]

@Frank Gaibler

That error is typical if you configured "nifi.web.http.host" property with a hostname not recognized as belonging to the host. Is it possible you changed bith or changed this one by mistake at one time?

-

Thanks,

Matt

[#6]

@Matt Clarke

Ah ok. That makes sense. I did paste the public IP in both at the time. The current config works with having the setting empty or to private IP. Thank you. Mistery solved. >_<

[#7]

Hi @Matt Clarke,

thank you very much for your fast response. I am not running nifi in a cluster. I am just trying to get an RPG communication to work from one local NiFi (my PC) to remote NiFi (Azure VM).

Yes, the "nifi.remote.input.host" property is what I am concerned about. The rest I believe was configured correctly from the get go for HTTP-traffic.

It works for PC to PC with setting it to my network IP. Running two NiFi on the same PC it works with localhost (Local PC is Windows).

However, I can't get it to work with the server.
Setting it to local IP has no effect. Setting it to the public IP (or *servername*.*region*.cloudapp.azure.com) will result in a crash of NiFi, shortly after startup.
I assume I would need the public IP. However I am not able to reach it with telnet or ping. Neither from outside, nor the VM itself. Therefore I assume it is a problem with my Azure or Ubuntu config. It actually doesnt matter for the RPG which of the two I use on my PC. (I also installed an Apache webserver and it to is reachable). What doesn't seem to work is to get the data from the public IP to the internal one and back out again.

The correct way it should work is to use the public IP for "nifi.remote.input.host", isn't it?

The Input/Output ports are updated correctly on start/stop. However files are not send and I do not get an error message. (I stopped myInput deliberatly to show, that different states are visualized).

Thanks,

Frank