Configuring ambari views on Kerberized Cluster

[#1]

Hi Folks,

In the kerberized cluster, we integrated AD for Ambari authentication. Using the AD users, I am able to login to Ambari. But when I log in by default it lands on the views. But When I click any of the views, I see an error.

500 Authentication requiredCollapse Stack Trace

org.apache.hadoop.security.AccessControlException: Authentication required at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:334)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)

While configuring the file view here are the properties I've used :

Settings:

WebHDFS Username ${username}

WebHDFS Authorization = auth=KERBEROS;proxyuser=admin

Cluster Configuration

Related to the cluster HDFS and name node details.

After Kerberization I created a user "ambari­-user/ambari-Host_name_here@KDCRealm.com

And also created a key tab, copied on the ambari -server machine.

Stopped Ambari server and then

$ambari­-server setup­security

Specified the keytab of the Ambari-user (newly created the User in KDC) and started the Ambari-Server.

Trying to access the Ambari -view but getting the above error.

Did any one face similar issue?

I am following the HDP documention section Configuring Ambari User Views with a Secure Cluster : http://hortonworks.com/wp-content/uploads/2015/04/AmbariUserViewsTechPreview_v1.pdf

Regards,

DP

[#2]

@Eric Walk, @Neeraj Sabharwal

I could access the File view but still facing the issues with Pig and Hive. Followed the steps of the documentation for Pig/Hive also.

While I am trying to create a new script on Pig. I get the following error.

java.net.UnknownHostException: hahdfs

java.net.UnknownHostException: hahdfs
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

For Hive:

java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS];

DP

[#3]

@Darpan Patel

Haven't tried setting that up in a NameNodeHA environment yet, but it seems that it is trying to resolve the reference to the NN Service Name in DNS and failing.

As for the Hive error, I'd suggest stopping ambari-server, doing a kdestroy for the user as which ambari-server runs and a kinit as the ambari-server user before starting it again.

[#4]

@Eric Walk

For Hive as per your suggestion : I stopped Ambari, did kdestroy, did kinit with the ambariserver keytab and then tried accessing the Hive page. But I still see the same error.

 Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "gateway/192.168.1.8"; destination host is: "NameNode1_Host":8020;


 H020 Could not establish connecton to gateway_Host:10000: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: 

[#5]

@Darpan Patel

I would double check those host names and that the ports are open.

[#6]

Hi @Darpan Patel great content.. I have given you a few points as a reward.

[#7]

Thanks Mark.

[#8]

@Darpan Patel

Pig view doesn’t seem to support NN HA. We encountered issues with PIG during our recent upgrade.

In order to fix this, We created 2 Pig views, one for each NN.

+@Predrag Minovic

[#9]

@Hemant Kumar @Predrag Minovic

I think this is not true for Non Kererberized cluster. I remember configuring Pig view for HA-ed cluster on HDP 2.3, and it was working fine. Though after Kerberization I did not check the Pig views. Yesterday when I checked all are breaking.

[#10]

@Hemant Kumar

@Predrag Minovic @Darpan Patel

Pig view Kerberos

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/section_pig_vi...

[#11]

@Darpan Patel I'm not sure have you set your Ambari principal correctly. If you use:

WebHDFS Authorization: auth=KERBEROS;proxyuser=admin

Then you need Ambari principal called admin/ambari-Host_name_here@KDCRealm.com

However, you said that you created: ambari­-user/ambari-Host_name_here@KDCRealm.com

Make sure that proxyuser name is matching the principal's user name. Then, you also need to add the following properties to your custom core-site.xml (assuming the proxyuser name is "admin") and restart HDFS.

hadoop.proxyuser.admin.groups=*
hadoop.proxyuser.admin.hosts=*

Also, to run Pig view you need to add webhcat.proxyuser.admin.groups=* and webhcat.proxyuser.admin.hosts=* to your webhcat-site.xml, and restart Hive. This should be enough to have your views running.

Regarding view other settings, as mentioned by others, use custom settings and set all fields referring to the latest documentation. It's also a good idea to switch, if you can, to the latest version of Ambari-2.1.2.1 (though 2.2 was released yesterday). If your NN is configured for HA then in Files and Hive view set:

WebHDFS FileSystem URI = webhdfs://nnhalabel:50070 where nnhalabel is the logical name of your NN.

We found that in 2.1.2.1 this settings does't work for the Pig view as @Hemant Kumar said. Finally, to be sure that views support NN HA, you can cause a failover of NNs using for example the "haadmin -failover" command. Regarding Pig view support for NN HA in a non-kerberized cluster we haven't tested that.

I hope this helps.