Support Questions

Find answers, ask questions, and share your expertise

in ambari accumulo not started

avatar
Contributor

its showing this error

 

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_master.py", line 24, in 
    AccumuloScript('master').execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_script.py", line 60, in start
    accumulo_service( self.component, action = 'start')
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_service.py", line 46, in accumulo_service
    user=params.accumulo_user
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo org.apache.accumulo.master.state.SetGoalState NORMAL' returned 1. 2019-08-22 12:31:01,352 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on hard system reset or power loss
2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper
2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds
2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds
2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:04,954 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 4.0 seconds
2019-08-22 12:31:08,958 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:08,958 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 8.0 seconds
2019-08-22 12:31:16,962 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:16,962 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 16.0 seconds
2019-08-22 12:31:32,975 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:32,975 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 32.0 seconds
2019-08-22 12:32:04,982 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:32:04,983 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 60.0 seconds
2019-08-22 12:33:05,009 [server.Accumulo] INFO : Connected to HDFS
2019-08-22 12:33:05,225 [start.Main] ERROR: Thread 'org.apache.accumulo.master.state.SetGoalState' died.
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/107244d1-c766-4a27-a60f-55a06565e4a6/masters/goal_state
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.accumulo.fate.zookeeper.ZooUtil.putData(ZooUtil.java:288)
	at org.apache.accumulo.fate.zookeeper.ZooUtil.putPersistentData(ZooUtil.java:267)
	at org.apache.accumulo.fate.zookeeper.ZooReaderWriter.putPersistentData(ZooReaderWriter.java:68)
	at org.apache.accumulo.master.state.SetGoalState.main(SetGoalState.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.accumulo.start.Main$2.run(Main.java:130)
	at java.lang.Thread.run(Thread.java:745)
 stdout:
2019-08-22 12:30:37,639 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-08-22 12:30:37,673 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-22 12:30:37,942 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-08-22 12:30:37,952 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-22 12:30:37,954 - Group['hdfs'] {}
2019-08-22 12:30:37,956 - Group['hadoop'] {}
2019-08-22 12:30:37,956 - Group['users'] {}
2019-08-22 12:30:37,957 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,959 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,960 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,961 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,963 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-22 12:30:37,964 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,966 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,967 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-22 12:30:37,969 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,970 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-22 12:30:37,972 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,973 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2019-08-22 12:30:37,975 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,976 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,978 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-22 12:30:37,979 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-22 12:30:37,980 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2019-08-22 12:30:37,996 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2019-08-22 12:30:37,997 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2019-08-22 12:30:37,999 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-22 12:30:38,000 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-22 12:30:38,001 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2019-08-22 12:30:38,012 - call returned (0, '1015')
2019-08-22 12:30:38,013 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2019-08-22 12:30:38,025 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] due to not_if
2019-08-22 12:30:38,026 - Group['hdfs'] {}
2019-08-22 12:30:38,027 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2019-08-22 12:30:38,028 - FS Type: HDFS
2019-08-22 12:30:38,028 - Directory['/etc/hadoop'] {'mode': 0755}
2019-08-22 12:30:38,059 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-08-22 12:30:38,060 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2019-08-22 12:30:38,087 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2019-08-22 12:30:38,094 - Skipping Execute[('setenforce', '0')] due to not_if
2019-08-22 12:30:38,094 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2019-08-22 12:30:38,097 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2019-08-22 12:30:38,099 - Changing owner for /var/run/hadoop from 1012 to root
2019-08-22 12:30:38,099 - Changing group for /var/run/hadoop from 1002 to root
2019-08-22 12:30:38,100 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2019-08-22 12:30:38,101 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2019-08-22 12:30:38,111 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2019-08-22 12:30:38,115 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2019-08-22 12:30:38,138 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:38,163 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-08-22 12:30:38,166 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2019-08-22 12:30:38,166 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2019-08-22 12:30:38,178 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:38,185 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2019-08-22 12:30:38,190 - Skipping unlimited key JCE policy check and setup since it is not required
2019-08-22 12:30:38,685 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-22 12:30:38,688 - Directory['/usr/hdp/current/accumulo-master/conf/server'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0700}
2019-08-22 12:30:38,690 - XmlConfig['accumulo-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/accumulo-master/conf/server', 'mode': 0600, 'configuration_attributes': {}, 'owner': 'accumulo', 'configurations': ...}
2019-08-22 12:30:38,711 - Generating config: /usr/hdp/current/accumulo-master/conf/server/accumulo-site.xml
2019-08-22 12:30:38,711 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-site.xml'] {'owner': 'accumulo', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0600, 'encoding': 'UTF-8'}
2019-08-22 12:30:38,812 - Directory['/var/run/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-08-22 12:30:38,812 - Creating directory Directory['/var/run/accumulo'] since it doesn't exist.
2019-08-22 12:30:38,813 - Changing owner for /var/run/accumulo from 0 to accumulo
2019-08-22 12:30:38,813 - Changing group for /var/run/accumulo from 0 to hadoop
2019-08-22 12:30:38,814 - Directory['/var/log/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-08-22 12:30:38,828 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-env.sh'] {'content': InlineTemplate(...), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:38,840 - Writing File['/usr/hdp/current/accumulo-master/conf/server/accumulo-env.sh'] because contents don't match
2019-08-22 12:30:38,841 - PropertiesFile['/usr/hdp/current/accumulo-master/conf/server/client.conf'] {'owner': 'accumulo', 'group': 'hadoop', 'properties': {'instance.zookeeper.host': u'gaian-lap386.com:2181', 'instance.name': u'hdp-accumulo-instance', 'instance.zookeeper.timeout': u'30s'}}
2019-08-22 12:30:38,852 - Generating properties file: /usr/hdp/current/accumulo-master/conf/server/client.conf
2019-08-22 12:30:38,853 - File['/usr/hdp/current/accumulo-master/conf/server/client.conf'] {'owner': 'accumulo', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-08-22 12:30:38,886 - Writing File['/usr/hdp/current/accumulo-master/conf/server/client.conf'] because contents don't match
2019-08-22 12:30:38,887 - File['/usr/hdp/current/accumulo-master/conf/server/log4j.properties'] {'content': ..., 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:38,977 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/auditLog.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:38,983 - File['/usr/hdp/current/accumulo-master/conf/server/auditLog.xml'] {'content': Template('auditLog.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,042 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/generic_logger.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,047 - File['/usr/hdp/current/accumulo-master/conf/server/generic_logger.xml'] {'content': Template('generic_logger.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,049 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/monitor_logger.xml'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,053 - File['/usr/hdp/current/accumulo-master/conf/server/monitor_logger.xml'] {'content': Template('monitor_logger.xml.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,054 - File['/usr/hdp/current/accumulo-master/conf/server/accumulo-metrics.xml'] {'content': StaticFile('accumulo-metrics.xml'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0644}
2019-08-22 12:30:39,058 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/tracers'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,060 - File['/usr/hdp/current/accumulo-master/conf/server/tracers'] {'content': Template('tracers.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,133 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/gc'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,136 - File['/usr/hdp/current/accumulo-master/conf/server/gc'] {'content': Template('gc.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,208 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/monitor'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,211 - File['/usr/hdp/current/accumulo-master/conf/server/monitor'] {'content': Template('monitor.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,313 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/slaves'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,316 - File['/usr/hdp/current/accumulo-master/conf/server/slaves'] {'content': Template('slaves.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,324 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/masters'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,327 - File['/usr/hdp/current/accumulo-master/conf/server/masters'] {'content': Template('masters.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,371 - TemplateConfig['/usr/hdp/current/accumulo-master/conf/server/hadoop-metrics2-accumulo.properties'] {'owner': 'accumulo', 'template_tag': None, 'group': 'hadoop'}
2019-08-22 12:30:39,383 - File['/usr/hdp/current/accumulo-master/conf/server/hadoop-metrics2-accumulo.properties'] {'content': Template('hadoop-metrics2-accumulo.properties.j2'), 'owner': 'accumulo', 'group': 'hadoop', 'mode': None}
2019-08-22 12:30:39,408 - HdfsResource['/user/accumulo'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'accumulo', 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp'], 'mode': 0700}
2019-08-22 12:30:39,413 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://gaian-lap386.com:50070/webhdfs/v1/user/accumulo?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmppQ8v48 2>/tmp/tmpMd4Xdu''] {'logoutput': None, 'quiet': False}
2019-08-22 12:30:47,961 - call returned (0, '')
2019-08-22 12:30:47,961 - get_user_call_output returned (0, u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16420,"group":"hdfs","length":0,"modificationTime":1565956748405,"owner":"accumulo","pathSuffix":"","permission":"700","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', u'')
2019-08-22 12:30:47,963 - HdfsResource['hdfs://gaian-lap386.com:8020/apps/accumulo'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'accumulo', 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp'], 'mode': 0700}
2019-08-22 12:30:47,970 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://gaian-lap386.com:50070/webhdfs/v1/apps/accumulo?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpmiTo34 2>/tmp/tmpPW_TqM''] {'logoutput': None, 'quiet': False}
2019-08-22 12:30:48,109 - call returned (0, '')
2019-08-22 12:30:48,110 - get_user_call_output returned (0, u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16421,"group":"hdfs","length":0,"modificationTime":1565956757742,"owner":"accumulo","pathSuffix":"","permission":"700","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', u'')
2019-08-22 12:30:48,111 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://gaian-lap386.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp']}
2019-08-22 12:30:48,114 - File['/var/lib/ambari-agent/tmp/pass'] {'content': InlineTemplate(...), 'owner': 'accumulo', 'group': 'hadoop', 'mode': 0600}
2019-08-22 12:30:48,114 - Writing File['/var/lib/ambari-agent/tmp/pass'] because it doesn't exist
2019-08-22 12:30:48,115 - Changing owner for /var/lib/ambari-agent/tmp/pass from 0 to accumulo
2019-08-22 12:30:48,115 - Changing group for /var/lib/ambari-agent/tmp/pass from 0 to hadoop
2019-08-22 12:30:48,115 - Changing permission for /var/lib/ambari-agent/tmp/pass from 644 to 600
2019-08-22 12:30:48,116 - Execute['cat /var/lib/ambari-agent/tmp/pass | ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo init --instance-name hdp-accumulo-instance --clear-instance-name >/var/log/accumulo/accumulo-init.out 2>/var/log/accumulo/accumulo-init.err'] {'logoutput': True, 'not_if': "ambari-sudo.sh su accumulo -l -s /bin/bash -c ' /usr/hdp/current/hadoop-client/bin/hadoop --config /usr/hdp/3.1.0.0-78/hadoop/conf fs -stat hdfs://gaian-lap386.com:8020/apps/accumulo/data'", 'user': 'accumulo'}
2019-08-22 12:30:53,525 - Skipping Execute['cat /var/lib/ambari-agent/tmp/pass | ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo init --instance-name hdp-accumulo-instance --clear-instance-name >/var/log/accumulo/accumulo-init.out 2>/var/log/accumulo/accumulo-init.err'] due to not_if
2019-08-22 12:30:53,526 - File['/var/lib/ambari-agent/tmp/pass'] {'action': ['delete']}
2019-08-22 12:30:53,542 - Deleting File['/var/lib/ambari-agent/tmp/pass']
2019-08-22 12:30:53,544 - Directory['/home/accumulo'] {'owner': 'accumulo', 'group': 'hadoop', 'recursive_ownership': True}
2019-08-22 12:30:53,548 - Execute['ACCUMULO_CONF_DIR=/usr/hdp/current/accumulo-master/conf/server /usr/hdp/current/accumulo-client/bin/accumulo org.apache.accumulo.master.state.SetGoalState NORMAL'] {'not_if': "ambari-sudo.sh su accumulo -l -s /bin/bash -c 'ls /var/run/accumulo/accumulo-accumulo-master.pid >/dev/null 2>&1 && ps `cat /var/run/accumulo/accumulo-accumulo-master.pid` >/dev/null 2>&1'", 'user': 'accumulo'}

Command failed after 1 tries
1 ACCEPTED SOLUTION

avatar
Master Mentor

@Manoj690 

Going through your logs I can see that the Namenode is in SAFE MODE, and in this case it won't allow you to change the status of any file in the cluster including the logs.


2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper
2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds
2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds
2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode

 

To resolve the issue can you do the following as hdfs user

$ hdfs dfsadmin -safemode get
Safe mode is OFF
The above is the desired output but if you get ON then proceed like below

First backup your FS Image & Edits

$ hdfs dfsadmin -saveNamespace

 

Then exit the safemode

$ hdfs dfsadmin -safemode leave

Once successful then revalidate

$ hdfs dfsadmin -safemode get

 

This time it should be off and you can now successfully restart the failed services from Ambari everything should succeed

HTH

View solution in original post

1 REPLY 1

avatar
Master Mentor

@Manoj690 

Going through your logs I can see that the Namenode is in SAFE MODE, and in this case it won't allow you to change the status of any file in the cluster including the logs.


2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper
2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds
2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode
2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds
2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode

 

To resolve the issue can you do the following as hdfs user

$ hdfs dfsadmin -safemode get
Safe mode is OFF
The above is the desired output but if you get ON then proceed like below

First backup your FS Image & Edits

$ hdfs dfsadmin -saveNamespace

 

Then exit the safemode

$ hdfs dfsadmin -safemode leave

Once successful then revalidate

$ hdfs dfsadmin -safemode get

 

This time it should be off and you can now successfully restart the failed services from Ambari everything should succeed

HTH