Disable Http Trace for Hbase master

[#1]

I am deploying hbase on yarn through apache slider. Once the hbase is up and running, if i check the Http TRACE is enabled on Hmaster or not with the command "curl -v -X TRACE hostname:port" , it is showing it to be enabled. So i tried to disable it by editing the web.xml with following changes:

<security-constraint>
<web-resource-collection>
<web-resource-name>Disable TRACE</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>TRACE</http-method>
</web-resource-collection>
<auth-constraint/>
</security-constraint>
<security-constraint>
<web-resource-collection>
<web-resource-name>Enable everything but TRACE</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method-omission>TRACE</http-method-omission>
</web-resource-collection>
</security-constraint>

But even this is not helping. It seems that any changes done in web.xml are not impacting the behaviour. So i checked online to disable Http trace then i got ""hbase.thrift.http.allow.options.method"" property for thrift server to disable Trace method. But i am not able to find anything for Hmaster. Could someone suggest the way to disable Trace method for Hbase master.

@Harsh J 

[#2]

What distribution and version of HBase are you using in your deployment
here?

On most recent CDH5 and CDH6 HBase Master web ports, performing a TRACE
method request responds with a 403 unauthorized response.

[#3]

I am using HBase 1.3.1 version of hbase.
Is there a way i can disable Trace method?

[#4]

That version should carry the changes required for constraining TRACE
(HBASE-10473)

I just tried standing a pseudo-distributed cluster over an Apache HBase
1.3.1 tarball, and it gives me the same result of a 403 when a TRACE
request is attempted.

Are you certain the end-point you're targeting is the HBase Master (does a
GET of /jmx over it return Master-metrics?), and can you share your full
curl -v output?

[#5]

Following are the commands that i am using :

1. Firstly i fetch Hmaster processId  by following command:

ps -ef | grep org.apache.hadoop.hbase.master.HMaster

2.Using the process id captured in first step i run the following command to fetch master ports:

netstat -tulpn | grep 24532
tcp6 0 0 :::35216 :::* LISTEN 24532/java
tcp6 0 0 :::46645 :::* LISTEN 24532/java
tcp6 0 0 10.65.157.61:19389 :::* LISTEN 24532/java

3. It is giving 3 ports. Curl O/p of first port is :

curl -v -X TRACE http://vmforpriyanka:35216
* About to connect() to vmforpriyanka port 35216 (#0)
* Trying 10.65.157.61...
* Connected to vmforpriyanka (10.65.157.61) port 35216 (#0)
> TRACE / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: vmforpriyanka:35216
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: message/http
< Content-Length: 83
< Server: Jetty(6.1.26)
<
TRACE / HTTP/1.1
User-Agent: curl/7.29.0
Host: vmforpriyanka:35216
Accept: */*
* Connection #0 to host vmforpriyanka left intact

Curl o/p of second port is:

curl -v -X TRACE http://vmforpriyanka:46645
* About to connect() to vmforpriyanka port 46645 (#0)
* Trying 10.65.157.61...
* Connected to vmforpriyanka (10.65.157.61) port 46645 (#0)
> TRACE / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: vmforpriyanka:46645
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Content-Length: 0
< Server: Jetty(6.1.26)
<
* Connection #0 to host vmforpriyanka left intact

Also regarding the Jira ticket https://issues.apache.org/jira/browse/HBASE-10473, is this Hbase rest server only or for Hmaster also? The mentioned InfoServer class does not exist at path /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/InfoServer.java in hbase 1.3.1. 

[#6]

Following are the steps i performed:

1. Firstly i run the command ps -ef | grep org.apache.hadoop.hbase.master.HMaster to get the process Id of Hmaster.

2.Then i ran netstat -tulpn | grep 24532 to fetch all the ports for Hmaster. This gave 3 ports for Hmaster:

tcp6 0 0 :::35216 :::* LISTEN 24532/java
tcp6 0 0 :::46645 :::* LISTEN 24532/java
tcp6 0 0 10.65.157.61:19389 :::* LISTEN 24532/java

3. Http Trace o/p for first port is curl -v -X TRACE http://vmforpriyanka:35216
* About to connect() to vmforpriyanka port 35216 (#0)
* Trying 10.65.157.61...
* Connected to vmforpriyanka (10.65.157.61) port 35216 (#0)
> TRACE / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: vmforpriyanka:35216
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: message/http
< Content-Length: 83
< Server: Jetty(6.1.26)
<
TRACE / HTTP/1.1
User-Agent: curl/7.29.0
Host: vmforpriyanka:35216
Accept: */*
* Connection #0 to host vmforpriyanka left intact

curl o/p for second port is 

curl -v -X TRACE http://vmforpriyanka:46645
* About to connect() to vmforpriyanka port 46645 (#0)
* Trying 10.65.157.61...
* Connected to vmforpriyanka (10.65.157.61) port 46645 (#0)
> TRACE / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: vmforpriyanka:46645
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Content-Length: 0
< Server: Jetty(6.1.26)
<
* Connection #0 to host vmforpriyanka left intact

Also regarding jira https://issues.apache.org/jira/browse/HBASE-10473, It seems that Inforserver.java does not even exist at the mentioned path

• /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/InfoServer.java. Although in hbase1.3.1 it does exist in hbase-server project. But that piece of code which is mentioned in the jirs is not there in the file.

[#7]

The file locations have changed over versions (Apache HBase underwent a
modular restructure into client/server code), but what's important is that
the constraint is still present in the moved sources and (seemingly, for
'46645' port) works.

What UI/service is served over 35216 by the HMaster process, if you try to
directly check via a browser/GET? I'm assuming 46645 serves the actual
HMaster UI instead?

[#8]

Yes the file location for InfoServer.java file is different in Hbase1.3.1 as it is at location org.apache.hadoop.hbase.http.InfoServer now. But this file does not have constraint method code in it anymore although it there for Hbase Rest Server. So is it possible that code to Constraint methods got removed for Hmaster in hbsae1.3.1 version.

Regarding the ports, when i try 35216  port on browser it gets redirected to 46645 port and gives Hmaster Infoserver UI.

Also , i tried to explicilty invoke HttpServerUtil.constrainHttpMethods() in Hmaster.putUpJettyServer() . And after that i bring up hbase then port 35216  is also giving 403. This seems fine as it is not giving 200 status anymore. But my concern is, i have checked that if Http TRACE method is diabled then it should gove 405 status not 403. So Is this change fine or not?

[#9]

Regarding 403 vs. 405 you're right on the specific difference, but all 4xx
errors pertain to denying client request in some form, so it is not
indicative of a feature not fully working.

Am not sure on what the 35216 port is (logs can help tell what's starting
it) - there should be no need for a forwarding port/proxy, but perhaps it
is the method of deployment doing that? Direct access to the HMaster port
appears to be denying TRACE operations in your tests.

[#10]

@Harsh J  On further investigation i found that there is one property "hbase.master.infoserver.redirect" which if set as false in hbase-site.xml will only expose two ports instead of 3(which were coming earlier as i mentioned in the above comment) .  One is TCP and other is HTTP web UI port (say P1). Here 3rd port (say P2) is not coming which was getting redirected to the main one. But in region server web ui there is Hmaster web UI link with the port P2, as P2 is not coming in this case so it showing -1 as the port. This means we can not go from region server to Hmaster directly. So i have 2 queries now:

1. What could be the side effect of setting hbase.master.infoserver.redirect property as false?

2. Currently we are no running Hbase in HA, so is it safe to use this property?

Floowing are the regionServer webUIs with the difference after setting the property to false and before that

After setting the property value to falseBefore setting the property value to false.