Created on 08-22-2019 11:12 PM - last edited on 08-23-2019 08:38 AM by VidyaSargur
its showing this error
stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_master.py", line 24, in AccumuloScript('master').execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_script.py", line 60, in start accumulo_service( self.component, action = 'start') File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_service.py", line 46, in accumulo_service user=params.accumulo_user File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run returns=self.resource.returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo org.apache.accumulo.master.state.SetGoalState NORMAL' returned 1. 2019-08-22 12:31:01,352 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on hard system reset or power loss 2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper 2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS 2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds 2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds 2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:31:04,954 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 4.0 seconds 2019-08-22 12:31:08,958 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:31:08,958 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 8.0 seconds 2019-08-22 12:31:16,962 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:31:16,962 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 16.0 seconds 2019-08-22 12:31:32,975 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:31:32,975 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 32.0 seconds 2019-08-22 12:32:04,982 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:32:04,983 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 60.0 seconds 2019-08-22 12:33:05,009 [server.Accumulo] INFO : Connected to HDFS 2019-08-22 12:33:05,225 [start.Main] ERROR: Thread 'org.apache.accumulo.master.state.SetGoalState' died. org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/107244d1-c766-4a27-a60f-55a06565e4a6/masters/goal_state at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.accumulo.fate.zookeeper.ZooUtil.putData(ZooUtil.java:288) at org.apache.accumulo.fate.zookeeper.ZooUtil.putPersistentData(ZooUtil.java:267) at org.apache.accumulo.fate.zookeeper.ZooReaderWriter.putPersistentData(ZooReaderWriter.java:68) at org.apache.accumulo.master.state.SetGoalState.main(SetGoalState.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.accumulo.start.Main$2.run(Main.java:130) at java.lang.Thread.run(Thread.java:745) stdout: 2019-08-22 12:30:37,639 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78 2019-08-22 12:30:37,673 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf 2019-08-22 12:30:37,942 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78 2019-08-22 12:30:37,952 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf 2019-08-22 12:30:37,954 - Group['hdfs'] {} 2019-08-22 12:30:37,956 - Group['hadoop'] {} 2019-08-22 12:30:37,956 - Group['users'] {} 2019-08-22 12:30:37,957 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,959 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,960 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,961 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,963 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None} 2019-08-22 12:30:37,964 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,966 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,967 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None} 2019-08-22 12:30:37,969 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,970 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None} 2019-08-22 12:30:37,972 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,973 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None} 2019-08-22 12:30:37,975 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,976 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,978 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-08-22 12:30:37,979 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2019-08-22 12:30:37,980 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2019-08-22 12:30:37,996 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2019-08-22 12:30:37,997 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2019-08-22 12:30:37,999 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2019-08-22 12:30:38,000 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2019-08-22 12:30:38,001 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {} 2019-08-22 12:30:38,012 - call returned (0, '1015') 2019-08-22 12:30:38,013 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2019-08-22 12:30:38,025 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] due to not_if 2019-08-22 12:30:38,026 - Group['hdfs'] {} 2019-08-22 12:30:38,027 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']} 2019-08-22 12:30:38,028 - FS Type: HDFS 2019-08-22 12:30:38,028 - Directory['/etc/hadoop'] {'mode': 0755} 2019-08-22 12:30:38,059 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2019-08-22 12:30:38,060 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} 2019-08-22 12:30:38,087 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2019-08-22 12:30:38,094 - Skipping Execute[('setenforce', '0')] due to not_if 2019-08-22 12:30:38,094 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'} 2019-08-22 12:30:38,097 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'} 2019-08-22 12:30:38,099 - Changing owner for /var/run/hadoop from 1012 to root 2019-08-22 12:30:38,099 - Changing group for /var/run/hadoop from 1002 to root 2019-08-22 12:30:38,100 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'} 2019-08-22 12:30:38,101 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'} 2019-08-22 12:30:38,111 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'} 2019-08-22 12:30:38,115 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'} 2019-08-22 12:30:38,138 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2019-08-22 12:30:38,163 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2019-08-22 12:30:38,166 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2019-08-22 12:30:38,166 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} 2019-08-22 12:30:38,178 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644} 2019-08-22 12:30:38,185 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} 2019-08-22 12:30:38,190 - Skipping unlimited key JCE policy check and setup since it is not required 2019-08-22 12:30:38,685 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf 2019-08-22 12:30:38,688 - Directory['/usr/hdp/current/accumulo-master/conf/server'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0700} 2019-08-22 12:30:38,690 - XmlConfig['accumulo-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/accumulo-master/conf/server', 'mode': 0600, 'configuration_attributes': {}, 'owner': 'accumulo', 'configurations': ...} 2019-08-22 12:30:38,711 - Generating config: /usr/hdp/current/accumulo-master/conf/server/accumulo-site.xml 2019-08-22 12:30:38,711 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-site.xml'] {'owner': 'accumulo', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0600, 'encoding': 'UTF-8'} 2019-08-22 12:30:38,812 - Directory['/var/run/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'} 2019-08-22 12:30:38,812 - Creating directory Directory['/var/run/accumulo'] since it doesn't exist. 2019-08-22 12:30:38,813 - Changing owner for /var/run/accumulo from 0 to accumulo 2019-08-22 12:30:38,813 - Changing group for /var/run/accumulo from 0 to hadoop 2019-08-22 12:30:38,814 - Directory['/var/log/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'} 2019-08-22 12:30:38,828 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-env.sh'] {'content': InlineTemplate(...), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644} 2019-08-22 12:30:38,840 - Writing File['/usr/hdp/current/accumulo-master/conf/server/accumulo-env.sh'] because contents don't match 2019-08-22 12:30:38,841 - PropertiesFile['/usr/hdp/current/accumulo-master/conf/server/client.conf'] {'owner': 'accumulo', 'group': 'hadoop', 'properties': {'instance.zookeeper.host': u'gaian-lap386.com:2181', 'instance.name': u'hdp-accumulo-instance', 'instance.zookeeper.timeout': u'30s'}} 2019-08-22 12:30:38,852 - Generating properties file: /usr/hdp/current/accumulo-master/conf/server/client.conf 2019-08-22 12:30:38,853 - File['/usr/hdp/current/accumulo-master/conf/server/client.conf'] {'owner': 'accumulo', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2019-08-22 12:30:38,886 - Writing File['/usr/hdp/current/accumulo-master/conf/server/client.conf'] because contents don't match 2019-08-22 12:30:38,887 - File['/usr/hdp/current/accumulo-master/conf/server/log4j.properties'] {'content': ..., 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644} 2019-08-22 12:30:38,977 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/auditLog.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:38,983 - File['/usr/hdp/current/accumulo-master/conf/server/auditLog.xml'] {'content': Template('auditLog.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,042 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/generic_logger.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:39,047 - File['/usr/hdp/current/accumulo-master/conf/server/generic_logger.xml'] {'content': Template('generic_logger.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,049 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/monitor_logger.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:39,053 - File['/usr/hdp/current/accumulo-master/conf/server/monitor_logger.xml'] {'content': Template('monitor_logger.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,054 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-metrics.xml'] {'content': StaticFile('accumulo-metrics.xml'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644} 2019-08-22 12:30:39,058 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/tracers'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:39,060 - File['/usr/hdp/current/accumulo-master/conf/server/tracers'] {'content': Template('tracers.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,133 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/gc'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:39,136 - File['/usr/hdp/current/accumulo-master/conf/server/gc'] {'content': Template('gc.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,208 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/monitor'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:39,211 - File['/usr/hdp/current/accumulo-master/conf/server/monitor'] {'content': Template('monitor.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,313 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/slaves'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:39,316 - File['/usr/hdp/current/accumulo-master/conf/server/slaves'] {'content': Template('slaves.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,324 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/masters'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:39,327 - File['/usr/hdp/current/accumulo-master/conf/server/masters'] {'content': Template('masters.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,371 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/hadoop-metrics2-accumulo.properties'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'} 2019-08-22 12:30:39,383 - File['/usr/hdp/current/accumulo-master/conf/server/hadoop-metrics2-accumulo.properties'] {'content': Template('hadoop-metrics2-accumulo.properties.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None} 2019-08-22 12:30:39,408 - HdfsResource['/user/accumulo'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'accumulo', 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp'], 'mode': 0700} 2019-08-22 12:30:39,413 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://gaian-lap386.com:50070/webhdfs/v1/user/accumulo?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmppQ8v48 2>/tmp/tmpMd4Xdu''] {'logoutput': None, 'quiet': False} 2019-08-22 12:30:47,961 - call returned (0, '') 2019-08-22 12:30:47,961 - get_user_call_output returned (0, u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16420,"group":"hdfs","length":0,"modificationTime":1565956748405,"owner":"accumulo","pathSuffix":"","permission":"700","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', u'') 2019-08-22 12:30:47,963 - HdfsResource['hdfs://gaian-lap386.com:8020/apps/accumulo'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'accumulo', 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp'], 'mode': 0700} 2019-08-22 12:30:47,970 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://gaian-lap386.com:50070/webhdfs/v1/apps/accumulo?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpmiTo34 2>/tmp/tmpPW_TqM''] {'logoutput': None, 'quiet': False} 2019-08-22 12:30:48,109 - call returned (0, '') 2019-08-22 12:30:48,110 - get_user_call_output returned (0, u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16421,"group":"hdfs","length":0,"modificationTime":1565956757742,"owner":"accumulo","pathSuffix":"","permission":"700","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', u'') 2019-08-22 12:30:48,111 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp']} 2019-08-22 12:30:48,114 - File['/var/lib/ambari-agent/tmp/pass'] {'content': InlineTemplate(...), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0600} 2019-08-22 12:30:48,114 - Writing File['/var/lib/ambari-agent/tmp/pass'] because it doesn't exist 2019-08-22 12:30:48,115 - Changing owner for /var/lib/ambari-agent/tmp/pass from 0 to accumulo 2019-08-22 12:30:48,115 - Changing group for /var/lib/ambari-agent/tmp/pass from 0 to hadoop 2019-08-22 12:30:48,115 - Changing permission for /var/lib/ambari-agent/tmp/pass from 644 to 600 2019-08-22 12:30:48,116 - Execute['cat /var/lib/ambari-agent/tmp/pass | ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo init --instance-name hdp-accumulo-instance --clear-instance-name >/var/log/accumulo/accumulo-init.out 2>/var/log/accumulo/accumulo-init.err'] {'logoutput': True, 'not_if': "ambari-sudo.sh su accumulo -l -s /bin/bash -c ' /usr/hdp/current/hadoop-client/bin/hadoop --config /usr/hdp/3.1.0.0-78/hadoop/conf fs -stat hdfs://gaian-lap386.com:8020/apps/accumulo/data'", 'user': 'accumulo'} 2019-08-22 12:30:53,525 - Skipping Execute['cat /var/lib/ambari-agent/tmp/pass | ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo init --instance-name hdp-accumulo-instance --clear-instance-name >/var/log/accumulo/accumulo-init.out 2>/var/log/accumulo/accumulo-init.err'] due to not_if 2019-08-22 12:30:53,526 - File['/var/lib/ambari-agent/tmp/pass'] {'action': ['delete']} 2019-08-22 12:30:53,542 - Deleting File['/var/lib/ambari-agent/tmp/pass'] 2019-08-22 12:30:53,544 - Directory['/home/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'recursive_ownership': True} 2019-08-22 12:30:53,548 - Execute['ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo org.apache.accumulo.master.state.SetGoalState NORMAL'] {'not_if': "ambari-sudo.sh su accumulo -l -s /bin/bash -c 'ls /var/run/accumulo/accumulo-accumulo-master.pid >/dev/null 2>&1 && ps `cat /var/run/accumulo/accumulo-accumulo-master.pid` >/dev/null 2>&1'", 'user': 'accumulo'} Command failed after 1 tries
Created 08-25-2019 12:20 PM
Going through your logs I can see that the Namenode is in SAFE MODE, and in this case it won't allow you to change the status of any file in the cluster including the logs.
2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper
2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds
2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds
2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
To resolve the issue can you do the following as hdfs user
$ hdfs dfsadmin -safemode get
Safe mode is OFF
The above is the desired output but if you get ON then proceed like below
First backup your FS Image & Edits
$ hdfs dfsadmin -saveNamespace
Then exit the safemode
$ hdfs dfsadmin -safemode leave
Once successful then revalidate
$ hdfs dfsadmin -safemode get
This time it should be off and you can now successfully restart the failed services from Ambari everything should succeed
HTH
Created 08-25-2019 12:20 PM
Going through your logs I can see that the Namenode is in SAFE MODE, and in this case it won't allow you to change the status of any file in the cluster including the logs.
2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper
2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds
2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds
2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
To resolve the issue can you do the following as hdfs user
$ hdfs dfsadmin -safemode get
Safe mode is OFF
The above is the desired output but if you get ON then proceed like below
First backup your FS Image & Edits
$ hdfs dfsadmin -saveNamespace
Then exit the safemode
$ hdfs dfsadmin -safemode leave
Once successful then revalidate
$ hdfs dfsadmin -safemode get
This time it should be off and you can now successfully restart the failed services from Ambari everything should succeed
HTH