Member since
12-21-2020
30
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 94 | 03-22-2021 05:14 AM | |
| 106 | 02-02-2021 12:07 AM |
04-16-2021
12:28 AM
1 Kudo
Hi @rubysimmons63 , Apache Falcon is used for Replicating HDFS Data and Hive tables from one cluster to another. You can use it for setting up Production <-> Disaster Recovery setup for your cluster. More info can be found at the official documentation site. As of June 2019, Apache Falcon project has moved into Apache Attic project. Thanks, Megh
... View more
04-12-2021
01:02 AM
Hi @Daming Xue , I haven't explored Auto-TLS yet, but will check it out. Thanks for sharing. Thanks, Megh
... View more
04-11-2021
05:40 AM
Hello Everyone, I'm facing a rather strange problem with my Ranger policies. I'm not able to save modifications to any policies once they get created. When I create a policy for the first time, it get created successfully. Subsequently, when I try to modify that policy, and click on save, the UI simply doesn't submit. Checked the Ranger Admin Logs as well, no entries. Any suggestions? Thanks, Megh
... View more
04-11-2021
05:11 AM
Hi @Daming Xue , I had tried this one as well. Putting mytruststore.jks file into /etc/ranger/usersync/conf/ directory. But since this is Cloudera, the file gets dropped at the next restart since for each restart the config is refreshed to the latest running pid. So it didn't work out. Thanks, Megh
... View more
04-11-2021
03:38 AM
Hi @Daming Xue , The link you've shared is broken. Can you please re-share? Thanks, Megh
... View more
04-10-2021
08:05 AM
Thanks for the suggestion. I've created a community post here . Unfortunately, I don't have any direct connections to any Hue commiters, but I'll wait and see if somebody provides an update on this. Thanks, Megh
... View more
04-10-2021
02:43 AM
Just to add on to this, how can I suggest this feature improvement to HUE community for adding support for remote clusters in distcp editor? Thanks, Megh
... View more
04-10-2021
02:42 AM
Hi @Daming Xue , I'm aware of the distcp command, but for that I need to give users access to the terminal which is something I want to avoid for security reasons. I want them to run their distcp jobs through a web UI. I hope this clarifies. Thanks, Megh
... View more
04-10-2021
02:39 AM
Hello Everyone, I've recently installed Ranger on CDP Private Cloud Base 7.1.5. For usersync, I'm connecting to my organization AD. For some reason, the usersync is throwing SSLHandshakeException and is not working. 2021-04-10 13:41:28,715 ERROR org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: LdapUserGroupBuilder.getUsers() failed with exception:
javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: simple bind failed: <AD Domain>:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]]
at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(AbstractLdapNamingEnumeration.java:237)
at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(AbstractLdapNamingEnumeration.java:189)
at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.getUsers(LdapUserGroupBuilder.java:435)
at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:325)
at org.apache.ranger.usergroupsync.UserGroupSync.syncUserGroup(UserGroupSync.java:100)
at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:55)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.naming.CommunicationException: simple bind failed: <AD Domain>:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]
at com.sun.jndi.ldap.LdapReferralContext.<init>(LdapReferralContext.java:96)
at com.sun.jndi.ldap.LdapReferralException.getReferralContext(LdapReferralException.java:151)
at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreReferrals(AbstractLdapNamingEnumeration.java:325)
at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(AbstractLdapNamingEnumeration.java:227)
... 6 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alert.createSSLException(Alert.java:131)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:353)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:296)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:291)
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:652)
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:471)
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:367)
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:376)
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:154)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1279)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1188)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:401)
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:808)
at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:75)
at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1093)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:450)
at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:423)
at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:359)
at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:214)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2895)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:348)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxFromUrl(LdapCtxFactory.java:225)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:189)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:152)
at com.sun.jndi.url.ldap.ldapURLContextFactory.getObjectInstance(ldapURLContextFactory.java:52)
at javax.naming.spi.NamingManager.getURLObject(NamingManager.java:601)
at javax.naming.spi.NamingManager.processURL(NamingManager.java:381)
at javax.naming.spi.NamingManager.processURLAddrs(NamingManager.java:361)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:333)
at com.sun.jndi.ldap.LdapReferralContext.<init>(LdapReferralContext.java:119)
... 9 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at sun.security.validator.Validator.validate(Validator.java:271)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:312)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:221)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:128)
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:636)
... 39 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 45 more
2021-04-10 13:41:28,718 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: LdapUserGroupBuilder.getUsers() user count: 0
2021-04-10 13:41:28,721 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: deltaSyncUserTime = 0 and highestdeltaSyncUserTime = 0
2021-04-10 13:41:28,721 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: deltaSyncGroupTime = 0 and highestdeltaSyncGroupTime = 0 I've imported the LDAPS Certificate to /usr/java/default/jre/lib/security/cacerts and the following property is set to this path. ranger.usersync.truststore.file = /usr/java/default/jre/lib/security/cacerts The surprising thing is my usersync LDAP URL is set as follows: ranger.usersync.ldap.url = ldaps://<AD Domain Controller Server1>:636 but in the error I'm getting "simple bind failed: <AD Domain>:636". With the same configuration for all other properties the Ranger Admin Authentication with AD works perfectly, but usersync is not happening. Things I've already tried: From this link, I tried adding -Djavax.net.ssl.trustStore=/<path to the cacert> in ranger-usersync-services.sh file. From this link, I've tried adding ranger.usersync.sink.impl.class property in my config. Experimented with User search/Group Search settings. Kindly add your suggestions. Thanks, Megh
... View more
04-10-2021
12:58 AM
Hi @Daming Xue , Thanks for your reply. The other cluster I have is HDP, so as far as I understand, using replication from CM won't work. In any case, I think the actual purpose of DistCp is for copying Data between different clusters. The reason I'm exploring a UI based alternative is that I don't want to give terminal access to users for distcp. Any other possibility? Thanks, Megh
... View more
04-09-2021
01:26 AM
Hello Everyone, Has anyone tested the Distcp editor available in HUE with CDP 7.1.5? Currently I see the below screen: In this, I'm not able to manually enter the source cluster path. As soon as I click on the source path text box, it opens a dialog box to select the path from the current cluster HDFS. Has anybody tested pulling data from another cluster? What config changes are required for getting this to work? Thanks, Megh
... View more
03-22-2021
05:14 AM
1 Kudo
Added the same configuration in "HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml " section and restarted the cluster and now the issue is resolved. Thanks, Megh
... View more
03-22-2021
01:38 AM
Hello Everyone, We have recently deployed CDP. I need to fetch data from one of my other cluster (HDFS HA-Enabled) to this cluster using distcp. For configuring the remote nameservice in my CDP cluster, I followed the steps given in this link and added the configuration properties to " HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml " section of HDFS configuration in Cloudera Manager. After this, I restarted the services and tried to run the following command from one of my hosts in the cluster: hadoop fs -ls hdfs://<remote-nameservice-name>/ But it throws this error: 21/03/22 13:59:34 WARN fs.FileSystem: Failed to initialize fileystem hdfs://<remote-nameservice-name>/: java.lang.IllegalArgumentException: java.net.UnknownHostException: <remote-nameservice-name>
-ls: java.net.UnknownHostException: <remote-nameservice-name>
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
[-cat [-ignoreCrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>]
[-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] [-e] <path> ...]
[-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>]
[-createSnapshot <snapshotDir> [<snapshotName>]]
[-deleteSnapshot <snapshotDir> <snapshotName>]
[-df [-h] [<path> ...]]
[-du [-s] [-h] [-v] [-x] <path> ...]
[-expunge [-immediate]]
[-find <path> ... <expression> ...]
[-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-getfacl [-R] <path>]
[-getfattr [-R] {-n name | -d} [-e en] <path>]
[-getmerge [-nl] [-skip-empty-file] <src> <localdst>]
[-head <file>]
[-help [cmd ...]]
[-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...]]
[-mkdir [-p] <path> ...]
[-moveFromLocal [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
[-moveToLocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>]
[-renameSnapshot <snapshotDir> <oldName> <newName>]
[-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setfattr {-n name [-v value] | -x name} <path>]
[-setrep [-R] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] [-s <sleep interval>] <file>]
[-test -[defsz] <path>]
[-text [-ignoreCrc] <src> ...]
[-touch [-a] [-m] [-t TIMESTAMP ] [-c] <path> ...]
[-touchz <path> ...]
[-truncate [-w] <length> <path> ...]
[-usage [cmd ...]]
Generic options supported are:
-conf <configuration file> specify an application configuration file
-D <property=value> define a value for a given property
-fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt <local|resourcemanager:port> specify a ResourceManager
-files <file1,...> specify a comma-separated list of files to be copied to the map reduce cluster
-libjars <jar1,...> specify a comma-separated list of jar files to be included in the classpath
-archives <archive1,...> specify a comma-separated list of archives to be unarchived on the compute machines
The general command line syntax is:
command [genericOptions] [commandOptions]
Usage: hadoop fs [generic options] -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...] I checked the hdfs-site.xml files on the host and it doesn't have the updated config. What might be the reason behind this? Why does Cloudera manager not manage the config change in the back end? . Thanks, Megh
... View more
03-10-2021
05:21 AM
Hello @Atahar , The Vulnerability ID is "http-options-method-enabled". I Don't want to disable HTTP and enable HTTPS, I want to disable "HTTP Options Method". Thanks, Megh
... View more
03-09-2021
02:16 AM
Hi @Atahar , Thanks for your reply. I'm actually looking for a property to disable HTTP options method as this is being flagged as a vulnerability by my internal Security team. Thanks, Megh
... View more
02-25-2021
02:06 AM
Hello Everyone, Is there a reference document for exhaustive list of properties that can be set within ambari.properties? I would like to check whether Ambari has an option for disabling HTTP OPTIONS method. Thanks, Megh
... View more
02-08-2021
03:09 AM
Hi @rguruvannagari , How does Ambari get the kadmin principal service ticket? Is it possible to get it by running a command in the shell? I'm trying to figure out a way of logging into the kadmin shell. My setup is integrated with AD as the KDC. I'm trying to kadmin -p myadminprincipal@MYREALM on the terminal but it says "kadmin: Database error! Required KADM5 principal missing while initializing kadmin interface". It makes me wonder how ambari server manages to create principals and keytabs with those same credentials. I even tried to decode the ambari-agent kerberos_common.py files to see the flow and there also it seems that Ambari Server indeed uses kadmin for the same. Would really appreciate any help in this regard. Thanks, Megh
... View more
02-04-2021
06:22 AM
Yoou have done a great help to me by deciding to share this! 🙂 Cheers!
... View more
02-03-2021
03:02 AM
Hi @GangWar , On the node this issue was occuring, somehow the default jdk folder name was java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64, so I assumed that it is 1.8.0.161, but after your suggestion I went into the node and issued "java -version" and to my surprise it was indeed openjdk version "1.8.0_252". Following your suggestion has fixed the issue for me. Thanks a lot! Megh
... View more
02-02-2021
10:50 PM
Hello Everyone, I've recently enabled Kerberos in my cluster and since then one of my datanodes is not able to connect to the namenode: I see these entries in the namenode logs: 2021-02-03 12:06:15,699 INFO ipc.Server (Server.java:saslProcess(1573)) - Auth successful for $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)
2021-02-03 12:06:15,700 INFO ipc.Server (Server.java:authorizeConnection(2235)) - Connection from <datanode-ip>:42328 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/<datanode-hostname>@<Realm-Name> (auth:PROXY) via $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)
2021-02-03 12:06:15,700 INFO ipc.Server (Server.java:doRead(1006)) - Socket Reader #1 for port 8020: readAndProcess from client <datanode-ip> threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User: $4E8100-MH1MCLUV65LO@<Realm-Name> is not allowed to impersonate dn/<datanode-hostname>@<Realm-Name>] From the datanode logs: 2021-02-03 12:14:33,806 WARN datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(225)) - Problem connecting to server: <namenode-hostname>/<namenode-ip>:8020 I've ensured that the hostnames are in lowercase and consistent in all the nodes. Telnet is also happening from the datanode to namenode hostname on 8020 port. Regenerating Keytabs and restarting everything also didn't work. Any other areas to look into? Thanks, Megh
... View more
02-02-2021
12:07 AM
Fixed this by changing my hostnames to lowercase. Earlier the hostnames were mixed case and researching online I found that Kerberos expects the hostnames to be lowercase and Realm name to be uppercase. changing the hostnames to all lowercase resolved the issue. Thanks, Megh
... View more
02-01-2021
04:03 AM
Figured out the keytab error. I had changed the encryption types and it was resulting in this error. I reverted those changes in the Kerberos configuration and it came clean. But after the entire process, now the services are not coming up. Firstly, it is getting stuck on starting the namenode, with safe mode not getting turned off automatically. If I leave the namenode manually, then it is showing the below error: stderr:
2021-02-01 17:18:32,891 - The NameNode is still in Safemode. Please be careful with commands that need Safemode OFF.
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 367, in <module>
NameNode().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 850, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 100, in start
upgrade_suspended=params.upgrade_suspended, env=env)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 226, in namenode
create_hdfs_directories()
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 293, in create_hdfs_directories
mode=0777,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 555, in action_create_on_execute
self.action_delayed("create")
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 552, in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name, self)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 279, in action_delayed
self._assert_valid()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 238, in _assert_valid
self.target_status = self._get_file_status(target)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 381, in _get_file_status
list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u : 'http://<namenode-hostname>:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'' returned status_code=403.
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403 org.apache.hadoop.security.authentication.client.AuthenticationException</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /webhdfs/v1/tmp. Reason:
<pre> org.apache.hadoop.security.authentication.client.AuthenticationException</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
</body>
</html>
stdout:
2021-02-01 16:55:36,478 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2021-02-01 16:55:36,639 - Stack Feature Version Info: stack_version=2.6, version=2.6.1.0-129, current_cluster_version=2.6.1.0-129 -> 2.6.1.0-129
2021-02-01 16:55:36,645 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2021-02-01 16:55:36,646 - Group['kms'] {}
2021-02-01 16:55:36,648 - Group['livy'] {}
2021-02-01 16:55:36,648 - Group['spark'] {}
2021-02-01 16:55:36,648 - Group['ranger'] {}
2021-02-01 16:55:36,648 - Group['hadoop'] {}
2021-02-01 16:55:36,648 - Group['users'] {}
2021-02-01 16:55:36,648 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,649 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,650 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2021-02-01 16:55:36,650 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,651 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2021-02-01 16:55:36,651 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']}
2021-02-01 16:55:36,652 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2021-02-01 16:55:36,652 - User['kms'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,653 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,653 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,654 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2021-02-01 16:55:36,654 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,655 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,655 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,656 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,656 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,657 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2021-02-01 16:55:36,657 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2021-02-01 16:55:36,659 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2021-02-01 16:55:36,667 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2021-02-01 16:55:36,667 - Group['hdfs'] {}
2021-02-01 16:55:36,667 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2021-02-01 16:55:36,668 - FS Type:
2021-02-01 16:55:36,668 - Directory['/etc/hadoop'] {'mode': 0755}
2021-02-01 16:55:36,679 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2021-02-01 16:55:36,680 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2021-02-01 16:55:36,696 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2021-02-01 16:55:36,706 - Skipping Execute[('setenforce', '0')] due to not_if
2021-02-01 16:55:36,707 - Directory['/hadoop/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2021-02-01 16:55:36,709 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2021-02-01 16:55:36,709 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2021-02-01 16:55:36,713 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2021-02-01 16:55:36,714 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}
2021-02-01 16:55:36,718 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2021-02-01 16:55:36,726 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2021-02-01 16:55:36,726 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2021-02-01 16:55:36,727 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2021-02-01 16:55:36,730 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2021-02-01 16:55:36,736 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2021-02-01 16:55:36,991 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2021-02-01 16:55:36,992 - Stack Feature Version Info: stack_version=2.6, version=2.6.1.0-129, current_cluster_version=2.6.1.0-129 -> 2.6.1.0-129
2021-02-01 16:55:37,011 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2021-02-01 16:55:37,023 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1}
2021-02-01 16:55:37,063 - checked_call returned (0, '2.6.1.0-129', '')
2021-02-01 16:55:37,071 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf stop namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'only_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2021-02-01 16:55:42,273 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete']}
2021-02-01 16:55:42,274 - Pid file /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid is empty or does not exist
2021-02-01 16:55:42,277 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2021-02-01 16:55:42,281 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2021-02-01 16:55:42,281 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2021-02-01 16:55:42,288 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
2021-02-01 16:55:42,288 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,295 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2021-02-01 16:55:42,301 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2021-02-01 16:55:42,301 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,306 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2021-02-01 16:55:42,306 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2021-02-01 16:55:42,312 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2021-02-01 16:55:42,313 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,317 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2021-02-01 16:55:42,323 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2021-02-01 16:55:42,323 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,328 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'configurations': ...}
2021-02-01 16:55:42,334 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2021-02-01 16:55:42,334 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,372 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2021-02-01 16:55:42,378 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2021-02-01 16:55:42,378 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2021-02-01 16:55:42,399 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'root'}
2021-02-01 16:55:42,403 - Directory['/hadoop/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2021-02-01 16:55:42,403 - Skipping setting up secure ZNode ACL for HFDS as it's supported only for NameNode HA mode.
2021-02-01 16:55:42,406 - Called service start with upgrade_type: None
2021-02-01 16:55:42,406 - Ranger Hdfs plugin is not enabled
2021-02-01 16:55:42,407 - File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'), 'group': 'hadoop'}
2021-02-01 16:55:42,407 - /hadoop/hadoop/hdfs/namenode/namenode-formatted/ exists. Namenode DFS already formatted
2021-02-01 16:55:42,407 - Directory['/hadoop/hadoop/hdfs/namenode/namenode-formatted/'] {'create_parents': True}
2021-02-01 16:55:42,407 - Options for start command are:
2021-02-01 16:55:42,408 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2021-02-01 16:55:42,408 - Changing owner for /var/run/hadoop from 0 to hdfs
2021-02-01 16:55:42,408 - Changing group for /var/run/hadoop from 0 to hadoop
2021-02-01 16:55:42,408 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2021-02-01 16:55:42,408 - Directory['/hadoop/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2021-02-01 16:55:42,409 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2021-02-01 16:55:42,417 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2021-02-01 16:55:46,632 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-<clustername>@<REALM>'] {'user': 'hdfs'}
2021-02-01 16:55:46,843 - Waiting for this NameNode to leave Safemode due to the following conditions: HA: False, isActive: True, upgradeType: None
2021-02-01 16:55:46,844 - Waiting up to 19 minutes for the NameNode to leave Safemode...
2021-02-01 16:55:46,844 - Execute['/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF''] {'logoutput': True, 'tries': 115, 'user': 'hdfs', 'try_sleep': 10}
safemode: Call From <namenode-hostname>/<namenode-ip> to <namenode-hostname>:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
2021-02-01 16:55:48,554 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From <namenode-hostname>/<namenode-ip> to <namenode-hostname>:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
2021-02-01 16:56:00,609 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:56:12,594 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:56:24,589 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:56:36,590 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:56:48,592 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:57:00,622 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:57:12,563 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:57:24,555 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:57:36,546 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:57:48,553 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:58:00,554 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:58:12,533 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:58:24,552 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:58:36,530 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:58:48,501 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:59:00,476 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:59:12,428 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:59:24,440 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:59:36,419 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 16:59:48,415 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:00:00,402 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:00:12,355 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:00:24,370 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:00:36,342 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:00:48,326 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:01:00,285 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:01:12,241 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:01:24,250 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:01:36,207 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:01:48,155 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:02:00,095 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:02:12,064 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:02:24,063 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:02:36,028 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:02:48,010 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:03:00,007 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:03:11,977 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:03:23,991 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:03:35,968 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:03:47,906 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:03:59,916 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:04:11,873 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:04:23,858 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:04:35,835 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:04:47,801 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:04:59,781 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:05:11,743 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:05:23,700 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:05:35,648 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:05:47,569 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:05:59,518 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:06:11,471 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:06:23,416 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:06:35,407 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:06:47,402 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:06:59,419 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:07:11,393 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:07:23,354 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:07:35,310 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:07:47,282 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:07:59,247 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:08:11,204 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:08:23,163 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:08:35,118 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:08:47,103 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:08:59,117 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:09:11,084 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:09:23,083 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:09:35,067 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:09:47,042 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:09:59,028 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:10:10,987 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:10:22,925 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:10:34,876 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:10:46,824 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:10:58,815 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:11:10,771 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:11:22,711 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:11:34,670 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:11:46,624 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:11:58,620 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:12:10,573 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:12:22,509 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:12:34,444 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:12:46,402 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:12:58,363 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:13:10,327 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:13:22,258 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:13:34,255 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:13:46,189 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:13:58,155 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:14:10,092 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:14:22,046 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:14:33,992 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:14:45,939 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:14:57,904 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:15:09,874 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:15:21,829 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:15:33,784 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:15:45,706 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:15:57,641 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:16:09,595 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:16:21,527 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:16:33,455 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:16:45,379 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:16:57,332 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:17:09,267 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:17:21,205 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:17:33,148 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:17:45,090 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:17:57,066 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:18:08,997 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:18:20,953 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://<namenode-hostname>:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
2021-02-01 17:18:32,891 - The NameNode is still in Safemode. Please be careful with commands that need Safemode OFF.
2021-02-01 17:18:32,891 - HdfsResource['/tmp'] {'security_enabled': True, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'dfs_type': '', 'default_fs': 'hdfs://<namenode-hostname>:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'hdfs-<clustername>@<REALM>', 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0777}
2021-02-01 17:18:32,893 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-<clustername>@<REALM>'] {'user': 'hdfs'}
2021-02-01 17:18:33,103 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u : '"'"'http://<namenode-hostname>:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp3JAW_l 2>/tmp/tmpksEFBR''] {'logoutput': None, 'quiet': False}
2021-02-01 17:18:33,291 - call returned (0, '')
Command failed after 1 tries Anybody can make sense of this? Thanks, Megh
... View more
01-27-2021
09:52 PM
Tried this today once again. Now the Keytab is generated but it doesn't contain any credentials. $ sudo -u ambari-qa klist -kte /etc/security/keytabs/kerberos.service_check.012821.keytab
Keytab name: FILE:/etc/security/keytabs/kerberos.service_check.012821.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------ The error log looks like this: stderr:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/KERBEROS/1.10.3-10/package/scripts/service_check.py", line 81, in <module>
KerberosServiceCheck().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/KERBEROS/1.10.3-10/package/scripts/service_check.py", line 64, in service_check
user=params.smoke_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/kerberos_service_check_cc_2d8eec0f439ec94bde482d7a8511a9c5 -kt /etc/security/keytabs/kerberos.service_check.012821.keytab bhimdr-012821@<EXAMPLE.COM>' returned 1. kinit: Keytab contains no suitable keys for bhimdr-012821@<EXAMPLE.COM> while getting initial credentials
stdout:
Performing kinit using bhimdr-012821@<EXAMPLE.COM>
2021-01-28 11:15:14,773 - Execute['/usr/bin/kinit -c /var/lib/ambari-agent/tmp/kerberos_service_check_cc_2d8eec0f439ec94bde482d7a8511a9c5 -kt /etc/security/keytabs/kerberos.service_check.012821.keytab bhimdr-012821@<EXAMPLE.COM>'] {'user': 'ambari-qa'}
2021-01-28 11:15:14,942 - File['/var/lib/ambari-agent/tmp/kerberos_service_check_cc_2d8eec0f439ec94bde482d7a8511a9c5'] {'action': ['delete']}
Command failed after 1 tries I have replaced my actual Realm name with <EXAMPLE.COM> in the above log. Thanks, Megh
... View more
01-27-2021
05:21 AM
Hello Everyone, I'm trying to enable kerberos on my HDP setup. during the wizard, it fails at Check Kerberos step. Before that, it shows that all the Keytabs creation and everything is getting successful. But the keytabs are not actually created. Please check the below screenshot: I checked in the host in question and the keytab file indeed didn't exist. Can anyone please give ideas where it might be going wrong? In the ambari-server log file I see the below error snippet. 27 Jan 2021 18:34:52,512 INFO [Server Action Executor Worker 6943] KerberosServerAction:353 - Processing identities...
27 Jan 2021 18:34:52,730 ERROR [Server Action Executor Worker 6943] FinalizeKerberosServerAction:114 - Failed to update the owner of the keytab file at /etc/security/keytabs/kerberos.service_check.012721.keytab to ambari-qa: chown: cannot access ‘/etc/security/keytabs/kerberos.service_check.012721.keytab’: No such file or directory
27 Jan 2021 18:34:52,738 ERROR [Server Action Executor Worker 6943] FinalizeKerberosServerAction:128 - Failed to update the group of the keytab file at /etc/security/keytabs/kerberos.service_check.012721.keytab to hadoop: chgrp: cannot access ‘/etc/security/keytabs/kerberos.service_check.012721.keytab’: No such file or directory
27 Jan 2021 18:34:52,745 ERROR [Server Action Executor Worker 6943] FinalizeKerberosServerAction:145 - Failed to update the access mode of the keytab file at /etc/security/keytabs/kerberos.service_check.012721.keytab to owner:'rw' and group:'rw': chmod: cannot access ‘/etc/security/keytabs/kerberos.service_check.012721.keytab’: No such file or directory
27 Jan 2021 18:34:52,746 INFO [Server Action Executor Worker 6943] KerberosServerAction:457 - Processing identities completed.
27 Jan 2021 18:34:53,746 INFO [qtp-ambari-agent-322] HeartBeatHandler:332 - HeartBeatHandler.sendCommands: sending ExecutionCommand for host checldbipba02.npci.org.in, role KERBEROS_SERVICE_CHECK, roleCommand SERVICE_CHECK, and command ID 546-5, task ID 6944
27 Jan 2021 18:34:55,522 ERROR [ambari-action-scheduler] ActionScheduler:447 - Operation completely failed, aborting request id: 546 Thanks, Megh
... View more
Labels:
01-27-2021
12:00 AM
Hi @abajwa, Does the Ambari Server Host also need to present it's own SSL certificate to the AD server? In case of multiple domain controllers, do we need to have separate SSL certificates from each of the domain controllers? Thanks, Megh
... View more
01-26-2021
11:58 PM
Hi @pcodding , Does the Ambari Server Host also need to present it's own SSL certificate to the AD server? In case of multiple domain controllers, do we need to have separate SSL certificates from each of the domain controllers? Thanks, Megh
... View more
01-26-2021
11:57 PM
Hi @sroberts , Does the Ambari Server Host also need to present it's own SSL certificate to the AD server? In case of multiple domain controllers, do we need to have separate SSL certificates from each of the domain controllers? Thanks, Megh
... View more
- Tags:
- Does
12-21-2020
09:09 PM
@smdas Thank you for the answer.
... View more
12-21-2020
04:24 AM
Hello All, I have an HDP 3.1 cluster that is neither kerberized nor it has ranger installed. Now, I have a requirement to Kerberize it and also enable Role Based Access Control using Ranger. 1. Should I first go with Kerberization and then installation of Ranger or vice versa? 2. Any cautions to be taken with both of these approaches? 3. Is there a requirement/recommended for implementing TLS/SSL for all the services while Kerberizing the cluster? Thank You, Megh Vidani
... View more
