NiFi 1.5 System Error: Invalid host header

[#1]

Hi Guys,

I just upgraded NiFi from v1.4 to v1.5. The cluster is set in secure mode and running in Kubernetes.

But I found below error in Web UI.

System Error The request contained an invalid host header [server:port] in the request [/nifi/]. Check for request manipulation or third-party intercept.

Is it due to nifi.web.proxy.host property? How do I setup it? the IP of the nifi node?

Thanks.

[#2]

@Alvin Jin

With NiFi 1.5, NiFi has become more restrictive with requard to the allowed headers coming from a client.

The hostname in the request header is checked against the configured hostname in the nifi.properties file:

nifi.web.http(s).host=

If they do not match, you will encounter the error you are seeing.

So you will need to access the NiFi UI in 1.5 using the same hostname as specified in that property.

Thanks,

Matt

[#3]

Hi @Matt Clarke,

Thank you for your response.

Is it nifi.web.http(s).host or nifi.web.proxy.host?

I have already put nifi.web.https.host = the-nifi-node-ip

We deploy nifi cluster in Kubernete, What should I change it to?

Thanks.

[#4]

@Alvin Jin

NiFi would have never started if a port was not specified. Let me be clear that you are setting up yoru NiFi to either run over a secure or non-secure port and not both.

for a non-secure NiFi set these:

nifi.web.http.host=
nifi.web.http.port=

and for a secured NiFi, set these:

nifi.web.https.host= 
nifi.web.https.port=

I am not familiar with Kubernete containers. Is the hostname of the container the same as the public facing hostname?

Thank you,

Matt

[#5]

Hi @Matt Clarke,

Thank you for your clarification. Yes, I use the secure mode.

In NiFi v1.4, I setup the two properties as below.

nifi.web.https.host= nifi-0.svc

nifi.web.https.port= 9443

Please note nifi-0.svc is the pod name of the first nifi node.

Then, we have nifi-1.svc, nifi-2.svc in the setting of the other nifi nodes.

Normally, if we want to access NiFi cluster outside of Kubernetes,

we have to use the master node ip of Kubernetes. e.g. https://master-ip:9443/nifi/

Above worked perfectly before.

Now my question is in NiFi v1.5. What should I set for nifi.web.https.host?

1. If I don't do any changes, it gives me "Invalid Host Header" error.

2. If I set it as nifi.web.https.host = master-ip in all three nifi nodes in the cluster.

The error in log is:

2018-01-18 17:10:28,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Cannot assign requested address

One thing confuses me is: I have three nifi nodes in a cluster. Should I set "nifi.web.https.host" in three nifi.property file to the same ip value?

BTW, are we able to disable this request header check?

Thanks.

[#6]

@Alvin Jin

Looks like you are hitting https://issues.apache.org/jira/browse/NIFI-4761

If you add a local host entry on your box that resolved "nifi-0.svc" to the IP associated to "master-ip", that should work if putting https://nifi-0.svc:9443/nifi/ does not work already.

Is that master node ip the same ip that is assigned to nifi-0.svc? If so, try setting the nifi.web.https.host= to the IP instead of the hostname.

Thanks,

Matt

[#7]

@Alvin Jin

The request header check cannot be disabled.

[#8]

Hi @Matt Clarke,

Kubernetes master ip is different with nifi-0.svc, which is an external pod dns.

So nifi-0.svc can't been access outside of Kubernetes.

I feel that nifi.web.proxy.host may work for this case.

Is there a way to disable the Request Header Check?

Thanks.

[#9]

The issue is that NiFi's Jetty server is being more restrictive about only accepting requests that were intended for its actual host.

So in your scenario, NiFi's Jetty server is listening on nifi-0.svc... then when you access https://master-ip:9443/nifi in your browser, the host header is set to master-ip, so when NiFi's Jetty server gets the request it rejects it because master-ip is not nifi-0.svc.

NIFI-4671 that Matt pointed out is going to add some way to white-list hostnames that should not be rejected, so you would just put master-ip in the white-list and it would let it through.

In the meantime, does Kubernetes have a way to control the headers that get forwarded from the master IP to the NiFi server?

For example, if someone had a load balancer in front of NiFi they would access the UI through the load balancer host which would create this same problem, but if you tell the load balancer to strip off the host header then it won't happen.

I don't know anything about Kubernetes so I have no idea if this is possible.

[#10]

NiFi 1.5 is validating "Host Header" and in Kubernetes the pod name is equal to host name. Unfortunately podname is generated during runtime and it is difficult to set in Host header.

Hortworks team, can you guys please help here to disable Host header check or work around of this issue.

,

It seems the check of "Host Header" of the request cannot be disabled. NiFi is expecting "pod" name of kubernetes as hostname in Host header. Unfortunately, It is not recommended to create "pod" explicitly it is created by "Deployment" and name of pod will be randomly generated during deployment. Is it possible for NiFi development team to provide some thought so that we can deploy in Kubernetes environment.

[#11]

I'm experiencing the same issue with running Docker 2 instances either locally on my laptop and on AWS.

Due to dockers setup, there should be some way to pass a startup parameter to:

- disable request header check

Locally Docker Nifi instance 1 : nifi1 on port 8080 is ok whilst on port 8090, where docker redirects, this fails.

This has stopped my current development.

Running a single instance without docker on AWS also generates this issue.

[#12]

This is so messed up, isn't it?

The external client request hostname and port has to match the internal hostname and port? I've lost a day to trying to figure out how to get that working when I have multiple nifi instances in docker-compose and I'm not getting anywhere.

The only thing I can think might work is adding a proxy to intercept and rewrite those, but that's insane out of the box behaviour, surely?

[#13]

Has anyone successfully come up with a non proxy based solution?

[#14]

I'm also getting this same error. Running Nifi v1.5 on an EC2 instance (AWS) and trying to access it from my browser as follows: ec2-instance-IP:port/nifi/.

Is there a way to allow this access? I don't need to disable the "header check" as a whole, but it would be great if I could somehow allow whatever header it is that I'm currently passing to be allowed to go through.

Any help is appreciated.

[#15 · Accepted solution]

@Vincent van Oudenhoven

A Jira was raised in Apache Jira (https://issues.apache.org/jira/browse/NIFI-4761) to add a whitelist capability so that users could create a list of hostnames that would be allowed. This Whitelist feature is already part of the HDF 3.1 release from Hortonworks and will be part of the Apache NiFi 1.6 release at a later time.

I am sorry to report there is no way around this host name check in Apache NIFi 1.5.

Thanks,

Matt

[#16]

@Matt Clarke

Thank you for your reply. Would it thus be advised to run NIFI 1.4 for now?

[#17]

Or you can run HDF 3.1, ambari makes it easy to manage your cluster. I recommend running Apache NiFi through HDF for a while.

[#18]

Thanks Matt. The whitelist is really helpful.

[#19]

Hello Vincent,

Could you please share the link to the Ambari Whitelist Configuration.

this only link I found in Ambari wiki is about the Ambari Metrics White list :

https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Metrics+-+Whitelisting

Best Regards

Abdou

[#20]

Hello @Matt Clarke,

Could you please share the link to the Ambari Whitelist Configuration ?

this only link I found in Ambari wiki is about the Ambari Metrics White list :

https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Metrics+-+Whitelisting

Best Regards

Abdou

[#21]

@Abdou B.

Not sure I follow. We are talking about whitelisting configuration needed for NiFi and not Ambari.

The specific nifi.properties property that is used to add a whitelist of allowed http headers is found in the NiFi admin guide:

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#web-properties

Thanks,

Matt