Re: Namenode UIs are not opening in any browser

pritam_konar · ‎09-16-2019

Can anyone please help?

pritam_konar · ‎09-11-2019

@jsensharma This problem is not solved and we are still unable to access Namenode UIs. In the last solution you mentioned to compare krb5.conf file from local cluster with that of hadoop cluster. But i could not find the krb5.conf file in hdfs.

pritam_konar · ‎08-30-2019

After re-installing hdf-select, i am getting the below error while setting the nifi version:- [root@nladfmrvu12 ~]# [root@nladfmrvu12 ~]# hdf-select File "/bin/hdf-select", line 216 print "ERROR: Invalid package - " + name ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Invalid package - " + name)? [root@nladfmrvu12 ~]# [root@nladfmrvu12 ~]# [root@nladfmrvu12 ~]# hdf-select set nifi 3.3.1.0-10 File "/bin/hdf-select", line 216 print "ERROR: Invalid package - " + name ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Invalid package - " + name)? [root@nladfmrvu12 ~]#

pritam_konar · ‎08-28-2019

I thing hdf-select is installed already in both the cluster. Please find the output below and confirm:- Please note , we installed Nifi via apache ambari. [root@nladfmrvu11 ~]# yum info hdf-select Loaded plugins: enabled_repos_upload, langpacks, package_upload, product-id, search-disabled-repos, subscription-manager Installed Packages Name : hdf-select Arch : noarch Version : 3.3.1.0 Release : 10 Size : 29 k Repo : installed From repo : HDF-3.3-repo-152 Summary : hdf-select Distro select package License : APL2 Description : hdf-select-3.3.1.0 select package Uploading Enabled Repositories Report Loaded plugins: langpacks, product-id, subscription-manager [root@nladfmrvu11 ~]# [root@nladfmrvu12 ~]# yum info hdf-select Loaded plugins: enabled_repos_upload, langpacks, package_upload, product-id, search-disabled-repos, subscription-manager Installed Packages Name : hdf-select Arch : noarch Version : 3.3.1.0 Release : 10 Size : 29 k Repo : installed From repo : HDF-3.3-repo-152 Summary : hdf-select Distro select package License : APL2 Description : hdf-select-3.3.1.0 select package Uploading Enabled Repositories Report Loaded plugins: langpacks, product-id, subscription-manager [root@nladfmrvu12 ~]# [root@nladfmrvu12 ~]#

pritam_konar · ‎08-28-2019

Thanks @jsensharma for the reply. I just want a confirmation:- If i run the command :- yum clean all will it remove all the installed yum packages, or it will remove only the 'hdf-select' packages?

pritam_konar · ‎08-27-2019

Nifi is installed in two servers. Please find the output from both of them:- 1) First Server:- [root@nladfmrvu11 ~]# hdf-select | grep nifi -bash: hdf-select: command not found [root@nladfmrvu11 ~]# [root@nladfmrvu11 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) [root@nladfmrvu11 ~]# [root@nladfmrvu11 ~]# ls -lart /usr/hdf total 4 drwxr-xr-x. 6 root root 66 Jul 29 15:21 3.3.1.0-10 drwxr-xr-x. 18 root root 4096 Aug 13 20:46 .. drwxr-xr-x. 5 root root 48 Aug 21 11:17 . drwxr-xr-x. 2 root root 17 Aug 21 11:18 None drwxr-xr-x. 2 root root 36 Aug 23 15:55 current [root@nladfmrvu11 ~]# [root@nladfmrvu11 ~]# ls -lart /usr/hdf/current/ total 0 lrwxrwxrwx. 1 root root 24 Jul 30 11:50 nifi -> /usr/hdf/3.3.1.0-10/nifi lrwxrwxrwx. 1 root root 32 Jul 30 11:52 nifi-toolkit -> /usr/hdf/3.3.1.0-10/nifi-toolkit drwxr-xr-x. 5 root root 48 Aug 21 11:17 .. drwxr-xr-x. 2 root root 36 Aug 23 15:55 . [root@nladfmrvu11 ~]# 2) Second Server:- [root@nladfmrvu12 ~]# hdf-select | grep nifi -bash: hdf-select: command not found [root@nladfmrvu12 ~]# [root@nladfmrvu12 ~]# [root@nladfmrvu12 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) [root@nladfmrvu12 ~]# [root@nladfmrvu12 ~]# ls -lart /usr/hdf total 4 drwxr-xr-x. 17 root root 4096 Aug 1 10:04 .. drwxr-xr-x. 5 root root 46 Aug 1 10:04 3.3.1.0-10 drwxr-xr-x. 2 root root 36 Aug 1 10:30 current drwxr-xr-x. 5 root root 48 Aug 23 11:41 . drwxr-xr-x. 2 root root 17 Aug 23 11:41 None [root@nladfmrvu12 ~]# ls -lart /usr/hdf/current/ total 0 lrwxrwxrwx. 1 root root 24 Aug 1 10:11 nifi -> /usr/hdf/3.3.1.0-10/nifi lrwxrwxrwx. 1 root root 32 Aug 1 10:12 nifi-toolkit -> /usr/hdf/3.3.1.0-10/nifi-toolkit drwxr-xr-x. 2 root root 36 Aug 1 10:30 . drwxr-xr-x. 5 root root 48 Aug 23 11:41 .. [root@nladfmrvu12 ~]#

pritam_konar · ‎08-26-2019

@Amir_haikh @kwabstian53 @jsensharma Please help here.

pritam_konar · ‎08-23-2019

Can anyone help here, please?

pritam_konar · ‎08-23-2019

When we are trying to implement Ranger Plugin in Nifi in our ambari clsuter, we are getting the attached error. We have configured Nifi in distributed mode in two different servers.Please find the attached error log. Ambari Version :- 2.7.3.0 HDP Version :- 3.1.0.0 HDF version :- 3.3.1.0-10 Nifi Version:- 1.8.0.3.3.1.0-10 stderr: 2019-08-22 17:08:51,422 - Stack selector path does not exist Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in <module> Master().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 144, in start setup_ranger_nifi(upgrade_type=None) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/setup_ranger_nifi.py", line 85, in setup_ranger_nifi component_user_keytab=params.ranger_nifi_keytab if params.security_enabled else None, cred_lib_path_override = cred_lib_prefix_path, cred_setup_prefix_override = cred_setup_prefix_path) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/setup_ranger_plugin_xml.py", line 74, in setup_ranger_plugin stack_version = get_stack_version(component_select_name) File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_stack_version.py", line 84, in get_stack_version 'Unable to determine the current version because of a non-zero return code of {0}'.format(str(return_code))) resource_management.core.exceptions.Fail: Unable to determine the current version because of a non-zero return code of 1

pritam_konar · ‎08-22-2019

Please find the information about edtuser:- [edtuser@nladfmrvu11 ~]$ [edtuser@nladfmrvu11 ~]$ klist klist: No credentials cache found (filename: /tmp/krb5cc_5009) [edtuser@nladfmrvu11 ~]$ [edtuser@nladfmrvu11 ~]$ ll /tmp/cache ls: cannot access /tmp/cache: No such file or directory [edtuser@nladfmrvu11 ~]$

pritam_konar · ‎08-22-2019

Hi, We are unable to access HDFS from a particular user 'edtuser' from putty after enabling kerberos. But we are able to access HDFS from 'root' user and 'hdfs' user. When we are trying to access HDFS from 'edtuser', the below error is coming:- [edtuser@nladfmrvu11 ~]$ [edtuser@nladfmrvu11 ~]$ hadoop fs -ls / 19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:06 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 1 failover attempts. Trying to failover after sleeping for 610ms. 19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:06 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 2 failover attempts. Trying to failover after sleeping for 1680ms. 19/08/22 09:48:08 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:08 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 3 failover attempts. Trying to failover after sleeping for 5206ms. 19/08/22 09:48:13 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:13 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 4 failover attempts. Trying to failover after sleeping for 10481ms. 19/08/22 09:48:24 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:24 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 5 failover attempts. Trying to failover after sleeping for 8347ms. 19/08/22 09:48:32 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:32 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 6 failover attempts. Trying to failover after sleeping for 13318ms. 19/08/22 09:48:45 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:45 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 7 failover attempts. Trying to failover after sleeping for 14341ms. 19/08/22 09:49:00 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:49:00 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 8 failover attempts. Trying to failover after sleeping for 17707ms. 19/08/22 09:49:17 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:49:17 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 9 failover attempts. Trying to failover after sleeping for 12086ms. 19/08/22 09:49:29 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:49:29 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 10 failover attempts. Trying to failover after sleeping for 11451ms. 19/08/22 09:49:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:49:41 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 11 failover attempts. Trying to failover after sleeping for 22482ms. 19/08/22 09:50:03 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:50:03 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 12 failover attempts. Trying to failover after sleeping for 22157ms. 19/08/22 09:50:25 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:50:25 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 13 failover attempts. Trying to failover after sleeping for 20978ms. 19/08/22 09:50:46 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:50:46 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 14 failover attempts. Trying to failover after sleeping for 16712ms. ^Z [1]+ Stopped hadoop fs -ls / [edtuser@nladfmrvu11 ~]$

pritam_konar · ‎08-22-2019

Hi, We are getting the below error whenever we are trying to access HDFS from console. This error is coming after enabling the kerberos authentication in the cluster. We are getting this error when we are trying to access from a customized user 'edtuser'. But we are not getting error when we are accessing error from 'root' user. and 'hdfs' user.Please find the error below:- [edtuser@nladfmrvu11 ~]$ [edtuser@nladfmrvu11 ~]$ hadoop fs -ls / 19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:06 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 1 failover attempts. Trying to failover after sleeping for 610ms. 19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:06 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 2 failover attempts. Trying to failover after sleeping for 1680ms. 19/08/22 09:48:08 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:08 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 3 failover attempts. Trying to failover after sleeping for 5206ms. 19/08/22 09:48:13 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:13 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 4 failover attempts. Trying to failover after sleeping for 10481ms. 19/08/22 09:48:24 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:24 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 5 failover attempts. Trying to failover after sleeping for 8347ms. 19/08/22 09:48:32 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:32 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 6 failover attempts. Trying to failover after sleeping for 13318ms. 19/08/22 09:48:45 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:48:45 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 7 failover attempts. Trying to failover after sleeping for 14341ms. 19/08/22 09:49:00 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:49:00 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 8 failover attempts. Trying to failover after sleeping for 17707ms. 19/08/22 09:49:17 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:49:17 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 9 failover attempts. Trying to failover after sleeping for 12086ms. 19/08/22 09:49:29 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:49:29 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 10 failover attempts. Trying to failover after sleeping for 11451ms. 19/08/22 09:49:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:49:41 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 11 failover attempts. Trying to failover after sleeping for 22482ms. 19/08/22 09:50:03 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:50:03 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 12 failover attempts. Trying to failover after sleeping for 22157ms. 19/08/22 09:50:25 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:50:25 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 13 failover attempts. Trying to failover after sleeping for 20978ms. 19/08/22 09:50:46 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/08/22 09:50:46 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 14 failover attempts. Trying to failover after sleeping for 16712ms. ^Z [1]+ Stopped hadoop fs -ls / [edtuser@nladfmrvu11 ~]$

pritam_konar · ‎08-22-2019

Hi, After enabling kerberos authentication in the cluster, we are not able to open any of the below mentioned namenode URLs in the browser ( for both active and standby namenocde) :- Namenode UI Namenode Logs Namenode JMX Thread Stacks Whenever we are trying to open the URLs, the below error is coming in the browser:- Please help us to resolve it.

pritam_konar · ‎08-06-2019

@Geoffrey Shelton Okot We regenerated the keytabs for all the services from Ambari and restarted all the services. Still we are getting the same error with Namenode restart. File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 295, in _run_command raise WebHDFSCallException(err_msg, result_dict) resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' --negotiate -u : 'http://nladfmrvu11.mtn.co.za:50070/webhdfs/v1/tmp?op=GETFILESTATUS'' returned status_code=401. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Authentication required</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /webhdfs/v1/tmp. Reason: <pre> Authentication required</pre></p> </body> </html> Please also let us know if we modify the below two properties in Advancecore-site like below , will the kerberos implementation be impacted :- hadoop.http.authentication.simple.anonymous.allowed=true hadoop.security.authentication=simple Please suggest.

pritam_konar · ‎08-05-2019

@ Geoffrey Shelton Okot Please note that we are setting this cluster in Linux servers. We are facing this issue after enabling Kerberos in the cluster. In our cluster the values of the above two properties are below. We need all the services up and running with kerberos implementation.

pritam_konar · ‎08-05-2019

We enabled kerberos in our Ambari Cluster ( Ambari version :- 2.7.3.0) with HDP version 3.1.0.0-78. But after kerberos implementation , we are getting the below error in HDFS and HBASE. Please suggest us to resolve the issue. File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 334, in _assert_valid self.target_status = self._get_file_status(target) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 497, in _get_file_status list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 214, in run_command return self._run_command(*args, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 295, in _run_command raise WebHDFSCallException(err_msg, result_dict) resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' --negotiate -u : 'http://nladfmrvu11.mtn.co.za:50070/webhdfs/v1/apps/hbase/data?op=GETFILESTATUS'' returned status_code=401. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Authentication required</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /webhdfs/v1/apps/hbase/data. Reason: <pre> Authentication required</pre></p> </body> </html>

pritam_konar · ‎07-30-2019

I re-created all the missing symlinks in the server for Nifi and it is working fine now. But when i am trying to install the Nifi in another server in the same cluster, I am getting the below error:- 2019-07-30 16:20:03,249 - Could not determine stack version for component nifi by calling '/usr/bin/hdf-select status nifi > /tmp/tmpLdzdzR'. Return Code: 1, Output: . 2019-07-30 16:20:03,276 - Could not determine stack version for component zookeeper-client by calling '/usr/bin/hdp-select status zookeeper-client > /tmp/tmpsVLHVn'. Return Code: 1, Output: .

pritam_konar · ‎07-30-2019

I am facing the below error while installing Nifi in the ambari cluster. Please find the log below:- Nifi.txt

pritam_konar · ‎07-23-2019

Is there any update available for this query?

pritam_konar · ‎07-04-2019

@Jay Kumar SenSharma Please find the latest log after commenting out the lines in build.py script. In ambari-metrics log, we can see the below error:- raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file)) resource_management.core.exceptions.Fail: Pid file /var/run/ambari-metrics-collector//hbase-ams-master.pid doesn't exist after starting of the component. I am attaching the full log of ambari-metrics:- Ambari_Metrics_log.txt

pritam_konar · ‎07-04-2019

@Gulshad Ansari and @Jay Kumar SenSharma .. any update in this problem?

pritam_konar · ‎07-03-2019

@Gulshad Ansari @Jay Kumar SenSharma Any other solution for this?

pritam_konar · ‎07-03-2019

@Jay Kumar SenSharma Based on your last comment, I commented out the line 40 in build.py. Still I am getting error for line 53 which was also a 'print' statement. When i commented out that line also, i am agian getting error for line 54. Please find the screenshot below:-

pritam_konar · ‎07-02-2019

@Gulshad Ansari @Jay Kumar SenSharma As per your latest comment, please find the answers below:- 1) Have you tried regenerating Keytabs from Ambari UI as mentioned in my previous update? Yes. I tried to regenerate the keytabs from Ambari UI. 2) Did you notice any error while generating missing keytabs? In Ambari Ui, I did not see any error. 3) Do you see that on the Problematic host the following file is present and has correct permission? 4) Are you able to get a valid kerberos ticket using that keytab on that host?

pritam_konar · ‎07-02-2019

Please find the output from the above command. The python script is throwing some syntactical error in the script:-

pritam_konar · ‎07-02-2019

@Jay Kumar SenSharma I followed the steps mentioned in the above Hortonworks documentation page. But when i tried to restart the services as a part of the kerberos setup, it is failing with the same error. Please find the attached log:- zookeeper_kerberos.txt

pritam_konar · ‎07-01-2019

This is a continuation of the below previous post :- https://community.hortonworks.com/questions/248489/error-in-smartsense.html After changing the configuration, i restarted the smartsense service, but it failed with attached error. SmartSense.txt I also checked the /var/log/smartsense-activity/activity-analyzer.log and the entry is like below:- 2019-07-01 15:31:09,983 INFO [main] ActivityUtil:412 - Could not find valid SmartSense ID. Will recheck every 5 minutes for next 5 minutes. 2019-07-01 15:36:09,987 ERROR [main] ActivityAnalyzerFacade:179 - Failed to analyze activities com.hortonworks.smartsense.activity.ActivityException: Invalid smartsense ID unspecified. Please configure a vaid SmartSense ID to proceed. at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:421) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:419) at com.hortonworks.smartsense.activity.util.ActivityUtil.validateSmartSenseId(ActivityUtil.java:396) at com.hortonworks.smartsense.activity.ActivityAnalyzerFacade.main(ActivityAnalyzerFacade.java:167) 2019-07-01 16:37:15,615 INFO [main] ActivityUtil:412 - Could not find valid SmartSense ID. Will recheck every 5 minutes for next 45 minutes. 2019-07-01 16:42:15,624 INFO [main] ActivityUtil:412 - Could not find valid SmartSense ID. Will recheck every 5 minutes for next 40 minutes. 2019-07-01 16:47:15,629 INFO [main] ActivityUtil:412 - Could not find valid SmartSense ID. Will recheck every 5 minutes for next 35 minutes.

pritam_konar · ‎07-01-2019

@Gulshad Ansari @Jay Kumar SenSharma After enabling the kerberos authentication, I am still getting the attached error in the log. In zookeeper log the error is:- File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-edtwinuat@MTN.CO.ZA;' returned 1. kinit: Client 'ambari-qa-edtwinuat@MTN.CO.ZA' not found in Kerberos database while getting initial credentials It seems that the keytabs are not properly setup in the cluster. Do i need to manually set these keytabs up? Please check the attached log for zookeeper and HDFS and let me know the solution?

pritam_konar · ‎07-01-2019

I tried with the above mentioned steps and tried to restart the ambari-metrics. But it is again giving the attached error:- ambari-metrics_log.txt

pritam_konar · ‎07-01-2019

@Gulshad Ansari @Jay Kumar SenSharma As the services are not running now, i disabled the kerberos and tried to re-enable it. Now i am getting the below error while re-enabling it. Please note that before kerberos enablement, HDFS was running normally. Error message: org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Failed to create the temporary data directory.

Online	Offline
Last Visited	‎09-16-2019 09:46 AM

Date Registered	‎06-04-2019 07:21 AM
Last Visited	‎09-16-2019 09:46 AM
Total Messages Posted	55

Re: Namenode UIs are not opening in any browser

Re: Namenode UIs are not opening in any browser

Re: Not able to configure Ranger plugin for Nifi

Re: Not able to configure Ranger plugin for Nifi

Re: Not able to configure Ranger plugin for Nifi

Re: Not able to configure Ranger plugin for Nifi

Re: Not able to configure Ranger plugin for Nifi

Re: Not able to configure Ranger plugin for Nifi

Not able to configure Ranger plugin for Nifi

Re: Not able tro access HDFS after enabling kerber...

HDFS is not accessible from an user after kerberos...

Not able tro access HDFS after enabling kerberos

Namenode UIs are not opening in any browser

Re: HDFS and HBASE are not starting after enabling...

Re: HDFS and HBASE are not starting after enabling...

HDFS and HBASE are not starting after enabling ker...

Re: Error while installing Nifi

Error while installing Nifi

Re: CSV file is not getting generated while enabli...

Re: Error in Ambari Metrics

Re: CSV file is not getting generated while enabli...

Re: CSV file is not getting generated while enabli...

Re: Error in Ambari Metrics

Re: CSV file is not getting generated while enabli...

Re: Error in Ambari Metrics

Re: CSV file is not getting generated while enabli...

Error in Smartsense

Re: CSV file is not getting generated while enabli...

Re: Error in Ambari Metrics

Re: CSV file is not getting generated while enabli...