Reply
Explorer
Posts: 7
Registered: ‎08-22-2018

Access Every Service UI using Old Hostname in New Cluster

We are migrating to new cluster. We performed HostName Redirection by alaising every old server in old cluster alias to every new server in new cluster. This way users will use the same old hostnames to connect to new cluster. 

 

I am currently testing HostName testing for every service that we have. Hostname testing for impala worked as expected when checked with impala-shell command. 

 

I am able to check all the jobs ran using Job History Server URL which has new server hostname. But when I give equivalent hostname that we mapped during hostname reidrection, it is saying this. 

 

HTTP ERROR 403

Problem accessing /jobhistory. Reason:

    GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)

 


Powered by Jetty://

 

Same is the case when I try to access Resource Manager and Namenode UI. 

Posts: 519
Topics: 14
Kudos: 91
Solutions: 45
Registered: ‎09-02-2016

Re: Access Every Service UI using Old Hostname in New Cluster

@Hichki

 

In general, this issue should go off after you kinit (if you are using command line).

 

This is just an additional check to make sure everything is good in your new cluster: Pls check whether the required kerbeors Principals are added as needed in the new cluster.  You can use the commands like kadmin.local and klist to get the principal and make sure the similar principals are available in your new cluster (CM -> Administration(menu) -> Security -> Kerberos Credentials). Pls make sure all the principals are mapped to the corresponding host in the above link

Posts: 1,002
Topics: 1
Kudos: 249
Solutions: 126
Registered: ‎04-22-2014

Re: Access Every Service UI using Old Hostname in New Cluster

@Hichki,

 

This  looks to be a server-side error response that can often be attributed to reverse DNS not being able to resolve to the same host the client made the connection to.

 

You can try running the following on your JobHistory Server host:

 

# getent hosts `hostname -f`

 

# getent hosts `hostname -f |awk '{print $0}'`

 

If the second command returns a different hostname, it is likely that is your issue.  Java Kerberos will attempt reverse DNS on the service principal host.  If the ip maps to a different hostname than, then the Service Ticket is essentially ignored.

 

MIT Kerberos allows reverse DNS to be turned off, but Java does not have a method at this time.

https://bugs.openjdk.java.net/browse/JDK-8189361

 

It is possible something else is causing the error, but you mentioned "aliasing" so I thought this may be relevant.

Highlighted
Explorer
Posts: 7
Registered: ‎08-22-2018

Re: Access Every Service UI using Old Hostname in New Cluster

Both commands returned same result.

 

We established hostname switch using DNS forwarding. I can see aliased old hostname when I do nslookup on new hostname. 

 

https:/<New_FQDN>:8090/cluster is the link where I am able to access new cluster Resource Manager Web UI. But it is giving above error when I access with old hostname https:/<Old_FQDN>:8090/cluster

 

And principals with old hostnames were already there in KDC but these are not showing up in Cloudera Manager. I am only seeing principals names generated with new hostnames. 

Announcements