question Re: NiFi 1.5 System Error: Invalid host header in Support Questions

NiFi 1.5 System Error: Invalid host header

alvinuw — Thu, 18 Jan 2018 22:31:12 GMT

Hi Guys,

I just upgraded NiFi from v1.4 to v1.5. The cluster is set in secure mode and running in Kubernetes.

But I found below error in Web UI.

System Error The request contained an invalid host header [server:port] in the request [/nifi/]. Check for request manipulation or third-party intercept.

Is it due to nifi.web.proxy.host property? How do I setup it? the IP of the nifi node?

Thanks.

Re: NiFi 1.5 System Error: Invalid host header

MattWho — Thu, 18 Jan 2018 23:08:59 GMT

@Alvin Jin

With NiFi 1.5, NiFi has become more restrictive with requard to the allowed headers coming from a client.

The hostname in the request header is checked against the configured hostname in the nifi.properties file:

nifi.web.http(s).host=

If they do not match, you will encounter the error you are seeing.

So you will need to access the NiFi UI in 1.5 using the same hostname as specified in that property.

Thanks,

Matt

Re: NiFi 1.5 System Error: Invalid host header

alvinuw — Fri, 19 Jan 2018 00:00:40 GMT

Hi @Matt Clarke,

Thank you for your response.

Is it nifi.web.http(s).host or nifi.web.proxy.host?

I have already put nifi.web.https.host = the-nifi-node-ip

We deploy nifi cluster in Kubernete, What should I change it to?

Thanks.

Re: NiFi 1.5 System Error: Invalid host header

MattWho — Fri, 19 Jan 2018 01:28:27 GMT

@Alvin Jin

NiFi would have never started if a port was not specified. Let me be clear that you are setting up yoru NiFi to either run over a secure or non-secure port and not both.

for a non-secure NiFi set these:

nifi.web.http.host=
nifi.web.http.port=

and for a secured NiFi, set these:

nifi.web.https.host= 
nifi.web.https.port=

I am not familiar with Kubernete containers. Is the hostname of the container the same as the public facing hostname?

Thank you,

Matt

Re: NiFi 1.5 System Error: Invalid host header

alvinuw — Fri, 19 Jan 2018 02:32:10 GMT

Hi @Matt Clarke,

Thank you for your clarification. Yes, I use the secure mode.

In NiFi v1.4, I setup the two properties as below.

nifi.web.https.host= nifi-0.svc

nifi.web.https.port= 9443

Please note nifi-0.svc is the pod name of the first nifi node.

Then, we have nifi-1.svc, nifi-2.svc in the setting of the other nifi nodes.

Normally, if we want to access NiFi cluster outside of Kubernetes,

we have to use the master node ip of Kubernetes. e.g. https://master-ip:9443/nifi/

Above worked perfectly before.

Now my question is in NiFi v1.5. What should I set for nifi.web.https.host?

1. If I don't do any changes, it gives me "Invalid Host Header" error.

2. If I set it as nifi.web.https.host = master-ip in all three nifi nodes in the cluster.

The error in log is:

2018-01-18 17:10:28,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Cannot assign requested address

One thing confuses me is: I have three nifi nodes in a cluster. Should I set "nifi.web.https.host" in three nifi.property file to the same ip value?

BTW, are we able to disable this request header check?

Thanks.

Re: NiFi 1.5 System Error: Invalid host header

MattWho — Fri, 19 Jan 2018 02:52:30 GMT

@Alvin Jin

Looks like you are hitting https://issues.apache.org/jira/browse/NIFI-4761

If you add a local host entry on your box that resolved "nifi-0.svc" to the IP associated to "master-ip", that should work if putting https://nifi-0.svc:9443/nifi/ does not work already.

Is that master node ip the same ip that is assigned to nifi-0.svc? If so, try setting the nifi.web.https.host= to the IP instead of the hostname.

Thanks,

Matt

Re: NiFi 1.5 System Error: Invalid host header

alvinuw — Fri, 19 Jan 2018 03:14:35 GMT

Hi @Matt Clarke,

Kubernetes master ip is different with nifi-0.svc, which is an external pod dns.

So nifi-0.svc can't been access outside of Kubernetes.

I feel that nifi.web.proxy.host may work for this case.

Is there a way to disable the Request Header Check?

Thanks.

Re: NiFi 1.5 System Error: Invalid host header

MattWho — Fri, 19 Jan 2018 05:30:05 GMT

@Alvin Jin

The request header check cannot be disabled.

Re: NiFi 1.5 System Error: Invalid host header

bbende — Fri, 19 Jan 2018 06:37:41 GMT

The issue is that NiFi's Jetty server is being more restrictive about only accepting requests that were intended for its actual host.

So in your scenario, NiFi's Jetty server is listening on nifi-0.svc... then when you access https://master-ip:9443/nifi in your browser, the host header is set to master-ip, so when NiFi's Jetty server gets the request it rejects it because master-ip is not nifi-0.svc.

NIFI-4671 that Matt pointed out is going to add some way to white-list hostnames that should not be rejected, so you would just put master-ip in the white-list and it would let it through.

In the meantime, does Kubernetes have a way to control the headers that get forwarded from the master IP to the NiFi server?

For example, if someone had a load balancer in front of NiFi they would access the UI through the load balancer host which would create this same problem, but if you tell the load balancer to strip off the host header then it won't happen.

I don't know anything about Kubernetes so I have no idea if this is possible.

Re: NiFi 1.5 System Error: Invalid host header

vinod_puchi — Thu, 25 Jan 2018 03:37:15 GMT

NiFi 1.5 is validating "Host Header" and in Kubernetes the pod name is equal to host name. Unfortunately podname is generated during runtime and it is difficult to set in Host header.

Hortworks team, can you guys please help here to disable Host header check or work around of this issue.

It seems the check of "Host Header" of the request cannot be disabled. NiFi is expecting "pod" name of kubernetes as hostname in Host header. Unfortunately, It is not recommended to create "pod" explicitly it is created by "Deployment" and name of pod will be randomly generated during deployment. Is it possible for NiFi development team to provide some thought so that we can deploy in Kubernetes environment.

Re: NiFi 1.5 System Error: Invalid host header

craig_knell — Sat, 27 Jan 2018 13:29:02 GMT

I'm experiencing the same issue with running Docker 2 instances either locally on my laptop and on AWS.

Due to dockers setup, there should be some way to pass a startup parameter to:

- disable request header check

Locally Docker Nifi instance 1 : nifi1 on port 8080 is ok whilst on port 8090, where docker redirects, this fails.

This has stopped my current development.

Running a single instance without docker on AWS also generates this issue.

Re: NiFi 1.5 System Error: Invalid host header

rich_midwinter — Fri, 23 Feb 2018 18:09:36 GMT

This is so messed up, isn't it?

The external client request hostname and port has to match the internal hostname and port? I've lost a day to trying to figure out how to get that working when I have multiple nifi instances in docker-compose and I'm not getting anywhere.

The only thing I can think might work is adding a proxy to intercept and rewrite those, but that's insane out of the box behaviour, surely?

Re: NiFi 1.5 System Error: Invalid host header

craig_knell — Sun, 25 Feb 2018 12:30:57 GMT

Has anyone successfully come up with a non proxy based solution?

Re: NiFi 1.5 System Error: Invalid host header

vincentvanouden — Mon, 26 Feb 2018 05:13:40 GMT

I'm also getting this same error. Running Nifi v1.5 on an EC2 instance (AWS) and trying to access it from my browser as follows: ec2-instance-IP:port/nifi/.

Is there a way to allow this access? I don't need to disable the "header check" as a whole, but it would be great if I could somehow allow whatever header it is that I'm currently passing to be allowed to go through.

Any help is appreciated.

Re: NiFi 1.5 System Error: Invalid host header

MattWho — Mon, 26 Feb 2018 21:04:00 GMT

@Vincent van Oudenhoven

A Jira was raised in Apache Jira (https://issues.apache.org/jira/browse/NIFI-4761) to add a whitelist capability so that users could create a list of hostnames that would be allowed. This Whitelist feature is already part of the HDF 3.1 release from Hortonworks and will be part of the Apache NiFi 1.6 release at a later time.

I am sorry to report there is no way around this host name check in Apache NIFi 1.5.

Thanks,

Matt

Re: NiFi 1.5 System Error: Invalid host header

vincentvanouden — Tue, 27 Feb 2018 08:30:24 GMT

@Matt Clarke

Thank you for your reply. Would it thus be advised to run NIFI 1.4 for now?

Re: NiFi 1.5 System Error: Invalid host header

TimothySpann — Tue, 27 Feb 2018 22:39:28 GMT

Thanks Matt. The whitelist is really helpful.

Re: NiFi 1.5 System Error: Invalid host header

TimothySpann — Tue, 27 Feb 2018 22:40:05 GMT

Or you can run HDF 3.1, ambari makes it easy to manage your cluster. I recommend running Apache NiFi through HDF for a while.

Re: NiFi 1.5 System Error: Invalid host header

tasanuma — Tue, 06 Mar 2018 13:58:50 GMT

If you can build NiFi, it is not so hard to back-port the whitelist feature to 1.5.0. You only need to cherry-pick two commits.

git clone -b rel/nifi-1.5.0 https://github.com/apache/nifi
cd nifi

# cherry-pick NIFI-4761
git cherry-pick 8cb09c301d6fef70cc8a02a4a7e80f3062ab58ae
git cherry-pick e3c661daac69cdb2de43c3d66d9ed1ccc9c8dbc6

# If you need the tar ball
mvn -T 2.0C clean install -DskipTests

# If you need the RPM package
mvn -T 2.0C clean install -DskipTests -Prpm

This works fine for us. Thanks.

Re: NiFi 1.5 System Error: Invalid host header

vincentvanouden — Wed, 07 Mar 2018 01:38:51 GMT

I'll make sure to try this out, I hadn't thought of doing this.