Configuring ambari views on Kerberized Cluster

[#1]

Hi Folks,

In the kerberized cluster, we integrated AD for Ambari authentication. Using the AD users, I am able to login to Ambari. But when I log in by default it lands on the views. But When I click any of the views, I see an error.

500 Authentication requiredCollapse Stack Trace

org.apache.hadoop.security.AccessControlException: Authentication required at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:334)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458) 
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)

While configuring the file view here are the properties I've used :

Settings:

WebHDFS Username ${username}

WebHDFS Authorization = auth=KERBEROS;proxyuser=admin

Cluster Configuration

Related to the cluster HDFS and name node details.

After Kerberization I created a user "ambari­-user/ambari-Host_name_here@KDCRealm.com

And also created a key tab, copied on the ambari -server machine.

Stopped Ambari server and then

$ambari­-server setup­security

Specified the keytab of the Ambari-user (newly created the User in KDC) and started the Ambari-Server.

Trying to access the Ambari -view but getting the above error.

Did any one face similar issue?

I am following the HDP documention section Configuring Ambari User Views with a Secure Cluster : http://hortonworks.com/wp-content/uploads/2015/04/AmbariUserViewsTechPreview_v1.pdf

Regards,

DP

[#2 · Accepted solution]

@Darpan Patel

http://docs.hortonworks.com/HDPDocuments/Ambari-2....

If the cluster your views will communicate with is Kerberos-enabled, you need to configure the Ambari Server instance(s) for Kerberos and be sure to configure the views to work with Kerberos.

[#3]

@Neeraj Sabharwal, @Eric Walk

Guys,

Some comments advocate that in HA , Ambari views have issues.

Are there limitations of PIG & HIVE Ambari Views that they cannot work with HDP cluster in High Availability ? Could you please confirm?

[#4]

@Darpan Patel This thread is getting offtrack from the original question. I don't see HA support for Pig and Hive yet. Please accept one of the answers to close the thread if anyone of the answers did help.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/_settings_and_...

[#5]

@Neeraj Sabharwal

I tried configuring Hive/PIG views as per the documentation.

If you confirm that in the Keberized cluster and NN Highly available PIG/HIVe views not supported then I will close the thread 🙂

Thank you very much.

[#6]

@Darpan Patel haha 🙂 Link

Kerberos - Yes

Hive - http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/section_kerber...

Pig - http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/section_pig_vi...

HA - No

😉

[#7]

So I had a bunch of trouble with these, here are some of the things to note:

1. When creating the view in Ambari don't use the "Local Ambari Managed Cluster" option, always use the custom when you have a kerberized cluster.
2. Definitely read the instructions carefully (i.e. this one: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_ambari_views_guide/content/section_pig_vi...) per @Neeraj Sabharwal.
3. Stop Ambari Server, do a kdestroy for the user ambari-server run as, do a kinit for the ambari user using it's proper keytab as the ambari linux user, then start ambari-server again. Do this procedure each time you restart Ambari Server.
4. For the pig view, there was a known issue where you needed to add: ,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar to your templeton.libjars for WebHCat (https://issues.apache.org/jira/browse/AMBARI-13096). Check your Ambari version...

[#8]

@Eric Walk Thank you for sharing these details. @jeff @Paul Codding

[#9]

No worries, I hope some of these things have been fixed since I went through this back in September (#4 should be resolved in Ambari 2.1.2). The Kdestroy/Kinit thing was definitely strange, never did work out why that was needed.

[#10]

Thanks will check and update in a few hours. 🙂

[#11]

@Eric Walk, @Neeraj Sabharwal

I could access the File view but still facing the issues with Pig and Hive. Followed the steps of the documentation for Pig/Hive also.

While I am trying to create a new script on Pig. I get the following error.

java.net.UnknownHostException: hahdfs

java.net.UnknownHostException: hahdfs
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

For Hive:

java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS];

DP

[#12]

@Darpan Patel

Haven't tried setting that up in a NameNodeHA environment yet, but it seems that it is trying to resolve the reference to the NN Service Name in DNS and failing.

As for the Hive error, I'd suggest stopping ambari-server, doing a kdestroy for the user as which ambari-server runs and a kinit as the ambari-server user before starting it again.

[#13]

@Eric Walk

For Hive as per your suggestion : I stopped Ambari, did kdestroy, did kinit with the ambariserver keytab and then tried accessing the Hive page. But I still see the same error.

 Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "gateway/192.168.1.8"; destination host is: "NameNode1_Host":8020;


 H020 Could not establish connecton to gateway_Host:10000: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: 

[#14]

@Darpan Patel

I would double check those host names and that the ports are open.

[#15]

Hi @Darpan Patel great content.. I have given you a few points as a reward.

[#16]

Thanks Mark.

[#17]

@Darpan Patel

Pig view doesn’t seem to support NN HA. We encountered issues with PIG during our recent upgrade.

In order to fix this, We created 2 Pig views, one for each NN.

+@Predrag Minovic

[#18]

@Hemant Kumar @Predrag Minovic

I think this is not true for Non Kererberized cluster. I remember configuring Pig view for HA-ed cluster on HDP 2.3, and it was working fine. Though after Kerberization I did not check the Pig views. Yesterday when I checked all are breaking.

[#19]

@Hemant Kumar

@Predrag Minovic @Darpan Patel

Pig view Kerberos

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/section_pig_vi...

[#20]

@Darpan Patel I'm not sure have you set your Ambari principal correctly. If you use:

WebHDFS Authorization: auth=KERBEROS;proxyuser=admin

Then you need Ambari principal called admin/ambari-Host_name_here@KDCRealm.com

However, you said that you created: ambari­-user/ambari-Host_name_here@KDCRealm.com

Make sure that proxyuser name is matching the principal's user name. Then, you also need to add the following properties to your custom core-site.xml (assuming the proxyuser name is "admin") and restart HDFS.

hadoop.proxyuser.admin.groups=*
hadoop.proxyuser.admin.hosts=*

Also, to run Pig view you need to add webhcat.proxyuser.admin.groups=* and webhcat.proxyuser.admin.hosts=* to your webhcat-site.xml, and restart Hive. This should be enough to have your views running.

Regarding view other settings, as mentioned by others, use custom settings and set all fields referring to the latest documentation. It's also a good idea to switch, if you can, to the latest version of Ambari-2.1.2.1 (though 2.2 was released yesterday). If your NN is configured for HA then in Files and Hive view set:

WebHDFS FileSystem URI = webhdfs://nnhalabel:50070 where nnhalabel is the logical name of your NN.

We found that in 2.1.2.1 this settings does't work for the Pig view as @Hemant Kumar said. Finally, to be sure that views support NN HA, you can cause a failover of NNs using for example the "haadmin -failover" command. Regarding Pig view support for NN HA in a non-kerberized cluster we haven't tested that.

I hope this helps.

[#21]

Thanks @Predrag Minovic

Indeed this is quite detailed. I've a user ambariserver and principal ambariserver/ambari_host_name@KDCRealm.com

I also verified following two properties are added in the custom core site.

hadoop.proxyuser.ambariserver.groups=*
hadoop.proxyuser.ambariserver.hosts=*

PIG/Hive view, I've added following two properties in the webhcat-site.xml

webhcat.proxyuser.ambariserver.groups=*
webhcat.proxyuser.ambariserver.hosts=*

Accessing the Hive View we see error.

H020 Could not establish connecton to HiveServer2_HOST:10000:org.apache.thrift.transport.TTransportException