question Re: Configuring ambari views on Kerberized Cluster in Archives of Support Questions (Read Only)

Configuring ambari views on Kerberized Cluster

darpan_be — Tue, 22 Dec 2015 01:56:13 GMT

Hi Folks,

In the kerberized cluster, we integrated AD for Ambari authentication. Using the AD users, I am able to login to Ambari. But when I log in by default it lands on the views. But When I click any of the views, I see an error.

500 Authentication requiredCollapse Stack Trace

org.apache.hadoop.security.AccessControlException: Authentication required at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:334)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)

While configuring the file view here are the properties I've used :

Settings:

WebHDFS Username ${username}
WebHDFS Authorization = auth=KERBEROS;proxyuser=admin

Cluster Configuration

Related to the cluster HDFS and name node details.

After Kerberization I created a user "ambari-user/ambari-Host_name_here@KDCRealm.com

And also created a key tab, copied on the ambari -server machine.

Stopped Ambari server and then

$ambari-server setupsecurity

Specified the keytab of the Ambari-user (newly created the User in KDC) and started the Ambari-Server.

Trying to access the Ambari -view but getting the above error.

Did any one face similar issue?

I am following the HDP documention section Configuring Ambari User Views with a Secure Cluster : http://hortonworks.com/wp-content/uploads/2015/04/AmbariUserViewsTechPreview_v1.pdf

Regards,

Re: Configuring ambari views on Kerberized Cluster

nsabharwal — Tue, 22 Dec 2015 02:13:01 GMT

@Darpan Patel

http://docs.hortonworks.com/HDPDocuments/Ambari-2....

If the cluster your views will communicate with is Kerberos-enabled, you need to configure the Ambari Server instance(s) for Kerberos and be sure to configure the views to work with Kerberos.

Re: Configuring ambari views on Kerberized Cluster

ewalk — Tue, 22 Dec 2015 02:40:15 GMT

So I had a bunch of trouble with these, here are some of the things to note:

When creating the view in Ambari don't use the "Local Ambari Managed Cluster" option, always use the custom when you have a kerberized cluster.
Definitely read the instructions carefully (i.e. this one: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_ambari_views_guide/content/section_pig_view_kerberos_config.html) per @Neeraj Sabharwal.
Stop Ambari Server, do a kdestroy for the user ambari-server run as, do a kinit for the ambari user using it's proper keytab as the ambari linux user, then start ambari-server again. Do this procedure each time you restart Ambari Server.
For the pig view, there was a known issue where you needed to add: ,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar to your templeton.libjars for WebHCat (https://issues.apache.org/jira/browse/AMBARI-13096). Check your Ambari version...

Re: Configuring ambari views on Kerberized Cluster

nsabharwal — Tue, 22 Dec 2015 02:44:04 GMT

@Eric Walk Thank you for sharing these details. @jeff @Paul Codding

Re: Configuring ambari views on Kerberized Cluster

ewalk — Tue, 22 Dec 2015 03:01:03 GMT

No worries, I hope some of these things have been fixed since I went through this back in September (#4 should be resolved in Ambari 2.1.2). The Kdestroy/Kinit thing was definitely strange, never did work out why that was needed.

Re: Configuring ambari views on Kerberized Cluster

mherring — Tue, 22 Dec 2015 03:04:26 GMT

Hi @Darpan Patel great content.. I have given you a few points as a reward.

Re: Configuring ambari views on Kerberized Cluster

darpan_be — Tue, 22 Dec 2015 03:22:07 GMT

Thanks Mark.

Re: Configuring ambari views on Kerberized Cluster

darpan_be — Tue, 22 Dec 2015 03:22:31 GMT

Thanks will check and update in a few hours. 🙂

Re: Configuring ambari views on Kerberized Cluster

darpan_be — Tue, 22 Dec 2015 08:09:59 GMT

@Eric Walk, @Neeraj Sabharwal

I could access the File view but still facing the issues with Pig and Hive. Followed the steps of the documentation for Pig/Hive also.

While I am trying to create a new script on Pig. I get the following error.

java.net.UnknownHostException: hahdfs

java.net.UnknownHostException: hahdfs
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

For Hive:

java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS];

Re: Configuring ambari views on Kerberized Cluster

hemant_dindi — Tue, 22 Dec 2015 12:57:48 GMT

@Darpan Patel

Pig view doesn’t seem to support NN HA. We encountered issues with PIG during our recent upgrade.

In order to fix this, We created 2 Pig views, one for each NN.

+@Predrag Minovic

Re: Configuring ambari views on Kerberized Cluster

darpan_be — Tue, 22 Dec 2015 18:30:41 GMT

@Hemant Kumar @Predrag Minovic

I think this is not true for Non Kererberized cluster. I remember configuring Pig view for HA-ed cluster on HDP 2.3, and it was working fine. Though after Kerberization I did not check the Pig views. Yesterday when I checked all are breaking.

Re: Configuring ambari views on Kerberized Cluster

ewalk — Tue, 22 Dec 2015 22:23:27 GMT

@Darpan Patel

Haven't tried setting that up in a NameNodeHA environment yet, but it seems that it is trying to resolve the reference to the NN Service Name in DNS and failing.

As for the Hive error, I'd suggest stopping ambari-server, doing a kdestroy for the user as which ambari-server runs and a kinit as the ambari-server user before starting it again.

Re: Configuring ambari views on Kerberized Cluster

darpan_be — Wed, 23 Dec 2015 01:52:52 GMT

@Eric Walk

For Hive as per your suggestion : I stopped Ambari, did kdestroy, did kinit with the ambariserver keytab and then tried accessing the Hive page. But I still see the same error.

 Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "gateway/192.168.1.8"; destination host is: "NameNode1_Host":8020;


 H020 Could not establish connecton to gateway_Host:10000: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused:

Re: Configuring ambari views on Kerberized Cluster

darpan_be — Wed, 23 Dec 2015 06:22:13 GMT

@Neeraj Sabharwal, @Eric Walk

Guys,

Some comments advocate that in HA , Ambari views have issues.

Are there limitations of PIG & HIVE Ambari Views that they cannot work with HDP cluster in High Availability ? Could you please confirm?

Re: Configuring ambari views on Kerberized Cluster

ewalk — Wed, 23 Dec 2015 06:28:15 GMT

@Darpan Patel

I would double check those host names and that the ports are open.

Re: Configuring ambari views on Kerberized Cluster

pminovic — Wed, 23 Dec 2015 17:28:35 GMT

@Darpan Patel I'm not sure have you set your Ambari principal correctly. If you use:

WebHDFS Authorization: auth=KERBEROS;proxyuser=admin

Then you need Ambari principal called admin/ambari-Host_name_here@KDCRealm.com

However, you said that you created: ambari-user/ambari-Host_name_here@KDCRealm.com

Make sure that proxyuser name is matching the principal's user name. Then, you also need to add the following properties to your custom core-site.xml (assuming the proxyuser name is "admin") and restart HDFS.

hadoop.proxyuser.admin.groups=*
hadoop.proxyuser.admin.hosts=*

Also, to run Pig view you need to add webhcat.proxyuser.admin.groups=* and webhcat.proxyuser.admin.hosts=* to your webhcat-site.xml, and restart Hive. This should be enough to have your views running.

Regarding view other settings, as mentioned by others, use custom settings and set all fields referring to the latest documentation. It's also a good idea to switch, if you can, to the latest version of Ambari-2.1.2.1 (though 2.2 was released yesterday). If your NN is configured for HA then in Files and Hive view set:

WebHDFS FileSystem URI = webhdfs://nnhalabel:50070 where nnhalabel is the logical name of your NN.

We found that in 2.1.2.1 this settings does't work for the Pig view as @Hemant Kumar said. Finally, to be sure that views support NN HA, you can cause a failover of NNs using for example the "haadmin -failover" command. Regarding Pig view support for NN HA in a non-kerberized cluster we haven't tested that.

I hope this helps.

Re: Configuring ambari views on Kerberized Cluster

darpan_be — Wed, 23 Dec 2015 22:51:59 GMT

Thanks @Predrag Minovic

Indeed this is quite detailed. I've a user ambariserver and principal ambariserver/ambari_host_name@KDCRealm.com

I also verified following two properties are added in the custom core site.

hadoop.proxyuser.ambariserver.groups=*
hadoop.proxyuser.ambariserver.hosts=*

PIG/Hive view, I've added following two properties in the webhcat-site.xml

webhcat.proxyuser.ambariserver.groups=*
webhcat.proxyuser.ambariserver.hosts=*

Accessing the Hive View we see error.

H020 Could not establish connecton to HiveServer2_HOST:10000:org.apache.thrift.transport.TTransportException

Re: Configuring ambari views on Kerberized Cluster

pminovic — Wed, 23 Dec 2015 23:06:28 GMT

Okay, what's the status of the Files view now? Can you now browse the files? Also try to restart ambari-server just in case.

Regarding Hive error, what's your Hive transport mode, binary or http? Only Hive view packaged with Ambari-2.1.2.1 (and I guess 2.2) supports http mode, old Ambari versions support only binary mode.

Re: Configuring ambari views on Kerberized Cluster

nsabharwal — Wed, 23 Dec 2015 23:18:48 GMT

@Hemant Kumar

@Predrag Minovic @Darpan Patel

Pig view Kerberos

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/section_pig_view_kerberos_config.html

Re: Configuring ambari views on Kerberized Cluster

nsabharwal — Mon, 19 Aug 2019 12:27:56 GMT

@Darpan Patel This thread is getting offtrack from the original question. I don't see HA support for Pig and Hive yet. Please accept one of the answers to close the thread if anyone of the answers did help.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/_settings_and_cluster_configuration.html