Created on
08-22-2019
11:12 PM
- last edited on
08-23-2019
08:38 AM
by
VidyaSargur
its showing this error
stderr:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_master.py", line 24, in
AccumuloScript('master').execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_script.py", line 60, in start
accumulo_service( self.component, action = 'start')
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_service.py", line 46, in accumulo_service
user=params.accumulo_user
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo org.apache.accumulo.master.state.SetGoalState NORMAL' returned 1. 2019-08-22 12:31:01,352 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on hard system reset or power loss
2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper
2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds
2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds
2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:04,954 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 4.0 seconds
2019-08-22 12:31:08,958 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:08,958 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 8.0 seconds
2019-08-22 12:31:16,962 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:16,962 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 16.0 seconds
2019-08-22 12:31:32,975 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:32,975 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 32.0 seconds
2019-08-22 12:32:04,982 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:32:04,983 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 60.0 seconds
2019-08-22 12:33:05,009 [server.Accumulo] INFO : Connected to HDFS
2019-08-22 12:33:05,225 [start.Main] ERROR: Thread 'org.apache.accumulo.master.state.SetGoalState' died.
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/107244d1-c766-4a27-a60f-55a06565e4a6/masters/goal_state
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.accumulo.fate.zookeeper.ZooUtil.putData(ZooUtil.java:288)
at org.apache.accumulo.fate.zookeeper.ZooUtil.putPersistentData(ZooUtil.java:267)
at org.apache.accumulo.fate.zookeeper.ZooReaderWriter.putPersistentData(ZooReaderWriter.java:68)
at org.apache.accumulo.master.state.SetGoalState.main(SetGoalState.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.accumulo.start.Main$2.run(Main.java:130)
at java.lang.Thread.run(Thread.java:745)
stdout:
2019-08-22 12:30:37,639 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-08-22 12:30:37,673 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-22 12:30:37,942 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-08-22 12:30:37,952 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-22 12:30:37,954 - Group['hdfs'] {}
2019-08-22 12:30:37,956 - Group['hadoop'] {}
2019-08-22 12:30:37,956 - Group['users'] {}
2019-08-22 12:30:37,957 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,959 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,960 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,961 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,963 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-22 12:30:37,964 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,966 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,967 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-22 12:30:37,969 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,970 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-22 12:30:37,972 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,973 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2019-08-22 12:30:37,975 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,976 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,978 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,979 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-22 12:30:37,980 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2019-08-22 12:30:37,996 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2019-08-22 12:30:37,997 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2019-08-22 12:30:37,999 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-22 12:30:38,000 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-22 12:30:38,001 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2019-08-22 12:30:38,012 - call returned (0, '1015')
2019-08-22 12:30:38,013 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2019-08-22 12:30:38,025 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] due to not_if
2019-08-22 12:30:38,026 - Group['hdfs'] {}
2019-08-22 12:30:38,027 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2019-08-22 12:30:38,028 - FS Type: HDFS
2019-08-22 12:30:38,028 - Directory['/etc/hadoop'] {'mode': 0755}
2019-08-22 12:30:38,059 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-08-22 12:30:38,060 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2019-08-22 12:30:38,087 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2019-08-22 12:30:38,094 - Skipping Execute[('setenforce', '0')] due to not_if
2019-08-22 12:30:38,094 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2019-08-22 12:30:38,097 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2019-08-22 12:30:38,099 - Changing owner for /var/run/hadoop from 1012 to root
2019-08-22 12:30:38,099 - Changing group for /var/run/hadoop from 1002 to root
2019-08-22 12:30:38,100 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2019-08-22 12:30:38,101 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2019-08-22 12:30:38,111 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2019-08-22 12:30:38,115 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2019-08-22 12:30:38,138 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:38,163 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-08-22 12:30:38,166 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2019-08-22 12:30:38,166 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2019-08-22 12:30:38,178 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:38,185 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2019-08-22 12:30:38,190 - Skipping unlimited key JCE policy check and setup since it is not required
2019-08-22 12:30:38,685 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-22 12:30:38,688 - Directory['/usr/hdp/current/accumulo-master/conf/server'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0700}
2019-08-22 12:30:38,690 - XmlConfig['accumulo-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/accumulo-master/conf/server', 'mode': 0600, 'configuration_attributes': {}, 'owner': 'accumulo', 'configurations': ...}
2019-08-22 12:30:38,711 - Generating config: /usr/hdp/current/accumulo-master/conf/server/accumulo-site.xml
2019-08-22 12:30:38,711 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-site.xml'] {'owner': 'accumulo', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0600, 'encoding': 'UTF-8'}
2019-08-22 12:30:38,812 - Directory['/var/run/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-08-22 12:30:38,812 - Creating directory Directory['/var/run/accumulo'] since it doesn't exist.
2019-08-22 12:30:38,813 - Changing owner for /var/run/accumulo from 0 to accumulo
2019-08-22 12:30:38,813 - Changing group for /var/run/accumulo from 0 to hadoop
2019-08-22 12:30:38,814 - Directory['/var/log/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-08-22 12:30:38,828 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-env.sh'] {'content': InlineTemplate(...), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:38,840 - Writing File['/usr/hdp/current/accumulo-master/conf/server/accumulo-env.sh'] because contents don't match
2019-08-22 12:30:38,841 - PropertiesFile['/usr/hdp/current/accumulo-master/conf/server/client.conf'] {'owner': 'accumulo', 'group': 'hadoop', 'properties': {'instance.zookeeper.host': u'gaian-lap386.com:2181', 'instance.name': u'hdp-accumulo-instance', 'instance.zookeeper.timeout': u'30s'}}
2019-08-22 12:30:38,852 - Generating properties file: /usr/hdp/current/accumulo-master/conf/server/client.conf
2019-08-22 12:30:38,853 - File['/usr/hdp/current/accumulo-master/conf/server/client.conf'] {'owner': 'accumulo', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-08-22 12:30:38,886 - Writing File['/usr/hdp/current/accumulo-master/conf/server/client.conf'] because contents don't match
2019-08-22 12:30:38,887 - File['/usr/hdp/current/accumulo-master/conf/server/log4j.properties'] {'content': ..., 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:38,977 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/auditLog.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:38,983 - File['/usr/hdp/current/accumulo-master/conf/server/auditLog.xml'] {'content': Template('auditLog.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,042 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/generic_logger.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,047 - File['/usr/hdp/current/accumulo-master/conf/server/generic_logger.xml'] {'content': Template('generic_logger.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,049 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/monitor_logger.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,053 - File['/usr/hdp/current/accumulo-master/conf/server/monitor_logger.xml'] {'content': Template('monitor_logger.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,054 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-metrics.xml'] {'content': StaticFile('accumulo-metrics.xml'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:39,058 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/tracers'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,060 - File['/usr/hdp/current/accumulo-master/conf/server/tracers'] {'content': Template('tracers.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,133 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/gc'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,136 - File['/usr/hdp/current/accumulo-master/conf/server/gc'] {'content': Template('gc.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,208 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/monitor'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,211 - File['/usr/hdp/current/accumulo-master/conf/server/monitor'] {'content': Template('monitor.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,313 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/slaves'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,316 - File['/usr/hdp/current/accumulo-master/conf/server/slaves'] {'content': Template('slaves.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,324 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/masters'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,327 - File['/usr/hdp/current/accumulo-master/conf/server/masters'] {'content': Template('masters.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,371 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/hadoop-metrics2-accumulo.properties'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,383 - File['/usr/hdp/current/accumulo-master/conf/server/hadoop-metrics2-accumulo.properties'] {'content': Template('hadoop-metrics2-accumulo.properties.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,408 - HdfsResource['/user/accumulo'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'accumulo', 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp'], 'mode': 0700}
2019-08-22 12:30:39,413 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://gaian-lap386.com:50070/webhdfs/v1/user/accumulo?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmppQ8v48 2>/tmp/tmpMd4Xdu''] {'logoutput': None, 'quiet': False}
2019-08-22 12:30:47,961 - call returned (0, '')
2019-08-22 12:30:47,961 - get_user_call_output returned (0, u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16420,"group":"hdfs","length":0,"modificationTime":1565956748405,"owner":"accumulo","pathSuffix":"","permission":"700","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', u'')
2019-08-22 12:30:47,963 - HdfsResource['hdfs://gaian-lap386.com:8020/apps/accumulo'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'accumulo', 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp'], 'mode': 0700}
2019-08-22 12:30:47,970 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://gaian-lap386.com:50070/webhdfs/v1/apps/accumulo?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpmiTo34 2>/tmp/tmpPW_TqM''] {'logoutput': None, 'quiet': False}
2019-08-22 12:30:48,109 - call returned (0, '')
2019-08-22 12:30:48,110 - get_user_call_output returned (0, u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16421,"group":"hdfs","length":0,"modificationTime":1565956757742,"owner":"accumulo","pathSuffix":"","permission":"700","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', u'')
2019-08-22 12:30:48,111 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp']}
2019-08-22 12:30:48,114 - File['/var/lib/ambari-agent/tmp/pass'] {'content': InlineTemplate(...), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0600}
2019-08-22 12:30:48,114 - Writing File['/var/lib/ambari-agent/tmp/pass'] because it doesn't exist
2019-08-22 12:30:48,115 - Changing owner for /var/lib/ambari-agent/tmp/pass from 0 to accumulo
2019-08-22 12:30:48,115 - Changing group for /var/lib/ambari-agent/tmp/pass from 0 to hadoop
2019-08-22 12:30:48,115 - Changing permission for /var/lib/ambari-agent/tmp/pass from 644 to 600
2019-08-22 12:30:48,116 - Execute['cat /var/lib/ambari-agent/tmp/pass | ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo init --instance-name hdp-accumulo-instance --clear-instance-name >/var/log/accumulo/accumulo-init.out 2>/var/log/accumulo/accumulo-init.err'] {'logoutput': True, 'not_if': "ambari-sudo.sh su accumulo -l -s /bin/bash -c ' /usr/hdp/current/hadoop-client/bin/hadoop --config /usr/hdp/3.1.0.0-78/hadoop/conf fs -stat hdfs://gaian-lap386.com:8020/apps/accumulo/data'", 'user': 'accumulo'}
2019-08-22 12:30:53,525 - Skipping Execute['cat /var/lib/ambari-agent/tmp/pass | ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo init --instance-name hdp-accumulo-instance --clear-instance-name >/var/log/accumulo/accumulo-init.out 2>/var/log/accumulo/accumulo-init.err'] due to not_if
2019-08-22 12:30:53,526 - File['/var/lib/ambari-agent/tmp/pass'] {'action': ['delete']}
2019-08-22 12:30:53,542 - Deleting File['/var/lib/ambari-agent/tmp/pass']
2019-08-22 12:30:53,544 - Directory['/home/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'recursive_ownership': True}
2019-08-22 12:30:53,548 - Execute['ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo org.apache.accumulo.master.state.SetGoalState NORMAL'] {'not_if': "ambari-sudo.sh su accumulo -l -s /bin/bash -c 'ls /var/run/accumulo/accumulo-accumulo-master.pid >/dev/null 2>&1 && ps `cat /var/run/accumulo/accumulo-accumulo-master.pid` >/dev/null 2>&1'", 'user': 'accumulo'}
Command failed after 1 tries
Created 08-25-2019 12:20 PM
Going through your logs I can see that the Namenode is in SAFE MODE, and in this case it won't allow you to change the status of any file in the cluster including the logs.
2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper
2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds
2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds
2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
To resolve the issue can you do the following as hdfs user
$ hdfs dfsadmin -safemode get
Safe mode is OFF
The above is the desired output but if you get ON then proceed like below
First backup your FS Image & Edits
$ hdfs dfsadmin -saveNamespace
Then exit the safemode
$ hdfs dfsadmin -safemode leave
Once successful then revalidate
$ hdfs dfsadmin -safemode get
This time it should be off and you can now successfully restart the failed services from Ambari everything should succeed
HTH
Created 08-25-2019 12:20 PM
Going through your logs I can see that the Namenode is in SAFE MODE, and in this case it won't allow you to change the status of any file in the cluster including the logs.
2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper
2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds
2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds
2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
To resolve the issue can you do the following as hdfs user
$ hdfs dfsadmin -safemode get
Safe mode is OFF
The above is the desired output but if you get ON then proceed like below
First backup your FS Image & Edits
$ hdfs dfsadmin -saveNamespace
Then exit the safemode
$ hdfs dfsadmin -safemode leave
Once successful then revalidate
$ hdfs dfsadmin -safemode get
This time it should be off and you can now successfully restart the failed services from Ambari everything should succeed
HTH