Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error when enabling namenode HA

Labels:
avatar
author-rank sherrine
Explorer

Created ‎10-29-2018 07:39 PM

Hi All,

I get the following error -

Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:136)
at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:187)

At the "Finalize HA Setup" stage

stderr: /var/lib/ambari-agent/data/errors-1233.txt 

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py", line 173, in <module>
    ZkfcSlave().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py", line 58, in start
    ZkfcSlaveDefault.start_static(env, upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py", line 84, in start_static
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 276, in service
    Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/2.6.5.0-292/hadoop/sbin/hadoop-daemon.sh --config /usr/hdp/2.6.5.0-292/hadoop/conf start zkfc'' returned 1. starting zkfc, logging to /var/log/hadoop/hdfs/hadoop-hdfs-zkfc-omiprihdp03ap.mufep.net.out
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:136)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:187)

stdout:

/var/lib/ambari-agent/data/output-1233.txt

2018-10-29 16:26:37,039 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.0-292 -> 2.6.5.0-292
2018-10-29 16:26:37,052 - Using hadoop conf dir: /usr/hdp/2.6.5.0-292/hadoop/conf
2018-10-29 16:26:37,215 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.0-292 -> 2.6.5.0-292
2018-10-29 16:26:37,219 - Using hadoop conf dir: /usr/hdp/2.6.5.0-292/hadoop/conf
2018-10-29 16:26:37,220 - Group['kms'] {}
2018-10-29 16:26:37,221 - Group['livy'] {}
2018-10-29 16:26:37,222 - Group['spark'] {}
2018-10-29 16:26:37,222 - Group['ranger'] {}
2018-10-29 16:26:37,222 - Group['hdfs'] {}
2018-10-29 16:26:37,222 - Group['zeppelin'] {}
2018-10-29 16:26:37,222 - Group['hadoop'] {}
2018-10-29 16:26:37,222 - Group['users'] {}
2018-10-29 16:26:37,223 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,224 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,225 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,225 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,226 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,227 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-10-29 16:26:37,228 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,229 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-10-29 16:26:37,229 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger'], 'uid': None}
2018-10-29 16:26:37,230 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-10-29 16:26:37,231 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'zeppelin', u'hadoop'], 'uid': None}
2018-10-29 16:26:37,232 - User['kms'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,233 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,234 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,234 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,235 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-10-29 16:26:37,236 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,237 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,238 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs'], 'uid': None}
2018-10-29 16:26:37,238 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,239 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,240 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,241 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,242 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-10-29 16:26:37,242 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-10-29 16:26:37,244 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-10-29 16:26:37,251 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-10-29 16:26:37,251 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-10-29 16:26:37,252 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-10-29 16:26:37,254 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-10-29 16:26:37,254 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2018-10-29 16:26:37,265 - call returned (0, '1030')
2018-10-29 16:26:37,266 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1030'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-10-29 16:26:37,272 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1030'] due to not_if
2018-10-29 16:26:37,273 - Group['hdfs'] {}
2018-10-29 16:26:37,273 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', u'hdfs']}
2018-10-29 16:26:37,274 - FS Type: 
2018-10-29 16:26:37,274 - Directory['/etc/hadoop'] {'mode': 0755}
2018-10-29 16:26:37,286 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-10-29 16:26:37,287 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-10-29 16:26:37,301 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-10-29 16:26:37,312 - Skipping Execute[('setenforce', '0')] due to not_if
2018-10-29 16:26:37,312 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-10-29 16:26:37,315 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-10-29 16:26:37,315 - Changing owner for /var/run/hadoop from 1020 to root
2018-10-29 16:26:37,315 - Changing group for /var/run/hadoop from 1000 to root
2018-10-29 16:26:37,315 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-10-29 16:26:37,319 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-10-29 16:26:37,320 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-10-29 16:26:37,325 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-10-29 16:26:37,333 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-10-29 16:26:37,334 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-10-29 16:26:37,334 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-10-29 16:26:37,337 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2018-10-29 16:26:37,342 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-10-29 16:26:37,620 - Using hadoop conf dir: /usr/hdp/2.6.5.0-292/hadoop/conf
2018-10-29 16:26:37,621 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.0-292 -> 2.6.5.0-292
2018-10-29 16:26:37,638 - Using hadoop conf dir: /usr/hdp/2.6.5.0-292/hadoop/conf
2018-10-29 16:26:37,650 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2018-10-29 16:26:37,655 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2018-10-29 16:26:37,655 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-10-29 16:26:37,661 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-policy.xml
2018-10-29 16:26:37,662 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-10-29 16:26:37,668 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-10-29 16:26:37,674 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/ssl-client.xml
2018-10-29 16:26:37,674 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-10-29 16:26:37,678 - Directory['/usr/hdp/2.6.5.0-292/hadoop/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-10-29 16:26:37,679 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2018-10-29 16:26:37,685 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/secure/ssl-client.xml
2018-10-29 16:26:37,685 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-10-29 16:26:37,689 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-10-29 16:26:37,695 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/ssl-server.xml
2018-10-29 16:26:37,695 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-10-29 16:26:37,701 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'configurations': ...}
2018-10-29 16:26:37,707 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/hdfs-site.xml
2018-10-29 16:26:37,707 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-10-29 16:26:37,744 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-10-29 16:26:37,750 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/core-site.xml
2018-10-29 16:26:37,750 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-10-29 16:26:37,772 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'}
2018-10-29 16:26:37,772 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.0-292 -> 2.6.5.0-292
2018-10-29 16:26:37,775 - Skipping setting up secure ZNode ACL for HFDS as it's supported only for secure clusters.
2018-10-29 16:26:37,775 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2018-10-29 16:26:37,775 - Changing owner for /var/run/hadoop from 0 to hdfs
2018-10-29 16:26:37,775 - Changing group for /var/run/hadoop from 0 to hadoop
2018-10-29 16:26:37,775 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2018-10-29 16:26:37,775 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2018-10-29 16:26:37,776 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2018-10-29 16:26:37,776 - File['/var/run/hadoop/hdfs/hadoop-hdfs-zkfc.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-zkfc.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-zkfc.pid'}
2018-10-29 16:26:37,800 - Deleting File['/var/run/hadoop/hdfs/hadoop-hdfs-zkfc.pid']
2018-10-29 16:26:37,801 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/2.6.5.0-292/hadoop/sbin/hadoop-daemon.sh --config /usr/hdp/2.6.5.0-292/hadoop/conf start zkfc''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/2.6.5.0-292/hadoop/libexec'}, 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-zkfc.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-zkfc.pid'}
2018-10-29 16:26:41,965 - Execute['find /var/log/hadoop/hdfs -maxdepth 1 -type f -name '*' -exec echo '==> {} <==' \; -exec tail -n 40 {} \;'] {'logoutput': True, 'ignore_failures': True, 'user': 'hdfs'}
==> /var/log/hadoop/hdfs/hadoop-hdfs-namenode-omiprihdp03ap.mufep.net.out.3 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201810240133 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(183554216k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T01:33:05.852+0200: 1.362: [GC (Allocation Failure) 2018-10-24T01:33:05.852+0200: 1.362: [ParNew: 209792K->24212K(235968K), 0.0630501 secs] 209792K->40598K(2070976K), 0.0631829 secs] [Times: user=0.44 sys=0.02, real=0.07 secs] 
2018-10-24T01:34:07.896+0200: 63.406: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 120652K(2070976K), 0.0079488 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2018-10-24T01:34:07.904+0200: 63.414: [CMS-concurrent-mark-start]
2018-10-24T01:34:07.908+0200: 63.418: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2018-10-24T01:34:07.908+0200: 63.418: [CMS-concurrent-preclean-start]
2018-10-24T01:34:07.914+0200: 63.424: [CMS-concurrent-preclean: 0.006/0.006 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2018-10-24T01:34:07.914+0200: 63.424: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T01:34:13.010+0200: 68.520: [CMS-concurrent-abortable-preclean: 1.216/5.096 secs] [Times: user=1.22 sys=0.00, real=5.09 secs] 
2018-10-24T01:34:13.010+0200: 68.520: [GC (CMS Final Remark) [YG occupancy: 104965 K (235968 K)]2018-10-24T01:34:13.010+0200: 68.520: [Rescan (parallel) , 0.0072251 secs]2018-10-24T01:34:13.017+0200: 68.527: [weak refs processing, 0.0000252 secs]2018-10-24T01:34:13.017+0200: 68.527: [class unloading, 0.0036285 secs]2018-10-24T01:34:13.021+0200: 68.531: [scrub symbol table, 0.0054746 secs]2018-10-24T01:34:13.026+0200: 68.536: [scrub string table, 0.0003966 secs][1 CMS-remark: 16386K(1835008K)] 121351K(2070976K), 0.0174924 secs] [Times: user=0.06 sys=0.00, real=0.02 secs] 
2018-10-24T01:34:13.028+0200: 68.538: [CMS-concurrent-sweep-start]
2018-10-24T01:34:13.028+0200: 68.539: [CMS-concurrent-sweep: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T01:34:13.028+0200: 68.539: [CMS-concurrent-reset-start]
2018-10-24T01:34:13.037+0200: 68.548: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 
2018-10-24T03:01:07.137+0200: 5282.647: [GC (Allocation Failure) 2018-10-24T03:01:07.137+0200: 5282.647: [ParNew: 234004K->7440K(235968K), 0.0530361 secs] 250390K->39189K(2070976K), 0.0531219 secs] [Times: user=0.31 sys=0.02, real=0.05 secs] 
2018-10-24T06:12:08.520+0200: 16744.030: [GC (Allocation Failure) 2018-10-24T06:12:08.520+0200: 16744.030: [ParNew: 217232K->2331K(235968K), 0.0068738 secs] 248981K->34081K(2070976K), 0.0069606 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2018-10-24T10:23:10.158+0200: 31805.668: [GC (Allocation Failure) 2018-10-24T10:23:10.158+0200: 31805.668: [ParNew: 212123K->1974K(235968K), 0.0066338 secs] 243873K->33724K(2070976K), 0.0067391 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 40729K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  18% used [0x0000000080000000, 0x00000000825d8ab0, 0x000000008cce0000)
  from space 26176K,   7% used [0x000000008cce0000, 0x000000008cecdab0, 0x000000008e670000)
  to   space 26176K,   0% used [0x000000008e670000, 0x000000008e670000, 0x0000000090000000)
 concurrent mark-sweep generation total 1835008K, used 31749K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 25564K, capacity 25848K, committed 26160K, reserved 1073152K
  class space    used 2709K, capacity 2810K, committed 2864K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-omiprihdp03ap.mufep.net.log <==
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:290)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:202)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:184)
	at com.sun.proxy.$Proxy11.rollEditLog(Unknown Source)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:522)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:405)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:371)
	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:476)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:367)
	at java.lang.Thread.run(Thread.java:748)
2018-10-29 11:31:00,115 INFO  namenode.SecondaryNameNode (SecondaryNameNode.java:run(465)) - Image has changed. Downloading updated image from NN.
2018-10-29 11:31:00,115 INFO  namenode.TransferFsImage (TransferFsImage.java:getFileClient(414)) - Opening connection to http://omiprihdp02ap.mufep.net:50070/imagetransfer?getimage=1&txid=191958&storageInfo=-63:732195773:...
2018-10-29 11:31:00,119 INFO  namenode.TransferFsImage (TransferFsImage.java:receiveFile(592)) - Combined time for fsimage download and fsync to all disks took 0.00s. The fsimage download took 0.00s at 138000.00 KB/s. Synchronous (fsync) write to disk of /hadoop/hdfs/namesecondary/current/fsimage.ckpt_0000000000000191958 took 0.00s.
2018-10-29 11:31:00,119 INFO  namenode.TransferFsImage (TransferFsImage.java:downloadImageToStorage(116)) - Downloaded file fsimage.ckpt_0000000000000191958 size 142071 bytes.
2018-10-29 11:31:00,120 INFO  namenode.TransferFsImage (TransferFsImage.java:getFileClient(414)) - Opening connection to http://omiprihdp02ap.mufep.net:50070/imagetransfer?getedit=1&startTxId=191959&endTxId=192022&storage...
2018-10-29 11:31:00,122 INFO  namenode.TransferFsImage (TransferFsImage.java:receiveFile(592)) - Combined time for fsimage download and fsync to all disks took 0.00s. The fsimage download took 0.00s at 8000.00 KB/s. Synchronous (fsync) write to disk of /hadoop/hdfs/namesecondary/current/edits_tmp_0000000000000191959-0000000000000192022_0000000000433577912 took 0.00s.
2018-10-29 11:31:00,122 INFO  namenode.TransferFsImage (TransferFsImage.java:downloadEditsToStorage(169)) - Downloaded file edits_tmp_0000000000000191959-0000000000000192022_0000000000433577912 size 0 bytes.
2018-10-29 11:31:00,133 INFO  namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSection(257)) - Loading 1744 INodes.
2018-10-29 11:31:00,140 INFO  namenode.FSImageFormatProtobuf (FSImageFormatProtobuf.java:load(184)) - Loaded FSImage in 0 seconds.
2018-10-29 11:31:00,140 INFO  namenode.FSImage (FSImage.java:loadFSImage(911)) - Loaded image for txid 191958 from /hadoop/hdfs/namesecondary/current/fsimage_0000000000000191958
2018-10-29 11:31:00,140 INFO  namenode.NameCache (NameCache.java:initialized(143)) - initialized with 3 entries 128 lookups
2018-10-29 11:31:00,141 INFO  namenode.Checkpointer (Checkpointer.java:rollForwardByApplyingLogs(313)) - Checkpointer about to load edits from 1 stream(s).
2018-10-29 11:31:00,141 INFO  namenode.FSImage (FSImage.java:loadEdits(849)) - Reading /hadoop/hdfs/namesecondary/current/edits_0000000000000191959-0000000000000192022 expecting start txid #191959
2018-10-29 11:31:00,141 INFO  namenode.FSImage (FSEditLogLoader.java:loadFSEdits(142)) - Start loading edits file /hadoop/hdfs/namesecondary/current/edits_0000000000000191959-0000000000000192022
2018-10-29 11:31:00,142 INFO  namenode.FSImage (FSEditLogLoader.java:loadFSEdits(145)) - Edits file /hadoop/hdfs/namesecondary/current/edits_0000000000000191959-0000000000000192022 of size 8272 edits # 64 loaded in 0 seconds
2018-10-29 11:31:00,142 INFO  namenode.FSImageFormatProtobuf (FSImageFormatProtobuf.java:save(417)) - Saving image file /hadoop/hdfs/namesecondary/current/fsimage.ckpt_0000000000000192022 using no compression
2018-10-29 11:31:00,149 INFO  namenode.FSImageFormatProtobuf (FSImageFormatProtobuf.java:save(421)) - Image file /hadoop/hdfs/namesecondary/current/fsimage.ckpt_0000000000000192022 of size 141653 bytes saved in 0 seconds .
2018-10-29 11:31:00,151 INFO  namenode.NNStorageRetentionManager (NNStorageRetentionManager.java:getImageTxIdToRetain(203)) - Going to retain 2 images with txid >= 191958
2018-10-29 11:31:00,151 INFO  namenode.NNStorageRetentionManager (NNStorageRetentionManager.java:purgeImage(225)) - Purging old image FSImageFile(file=/hadoop/hdfs/namesecondary/current/fsimage_0000000000000182244, cpktTxId=0000000000000182244)
2018-10-29 11:31:00,152 INFO  namenode.NNStorageRetentionManager (NNStorageRetentionManager.java:purgeImage(225)) - Purging old image FSImageFile(file=/hadoop/hdfs/namesecondary/current/fsimage_0000000000000172464, cpktTxId=0000000000000172464)
2018-10-29 11:31:00,157 INFO  namenode.TransferFsImage (TransferFsImage.java:copyFileToStream(395)) - Sending fileName: /hadoop/hdfs/namesecondary/current/fsimage_0000000000000192022, fileSize: 141653. Sent total: 141653 bytes. Size of last segment intended to send: -1 bytes.
2018-10-29 11:31:00,163 INFO  namenode.TransferFsImage (TransferFsImage.java:uploadImageFromStorage(238)) - Uploaded image with txid 192022 to namenode at http://omiprihdp02ap.mufep.net:50070 in 0.008 seconds
2018-10-29 11:31:00,163 WARN  namenode.SecondaryNameNode (SecondaryNameNode.java:doCheckpoint(576)) - Checkpoint done. New Image Size: 141653
2018-10-29 11:52:31,427 ERROR namenode.SecondaryNameNode (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2018-10-29 11:52:31,429 INFO  namenode.SecondaryNameNode (LogAdapter.java:info(45)) - SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at omiprihdp03ap.mufep.net/10.6.7.23
************************************************************/
==> /var/log/hadoop/hdfs/SecurityAuth.audit <==
==> /var/log/hadoop/hdfs/hdfs-audit.log <==
==> /var/log/hadoop/hdfs/gc.log-201810291544 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184342048k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T15:44:22.568+0200: 1.253: [GC (Allocation Failure) 2018-10-29T15:44:22.568+0200: 1.254: [ParNew: 209792K->14136K(235968K), 0.0130374 secs] 209792K->14136K(2070976K), 0.0137064 secs] [Times: user=0.05 sys=0.01, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 96401K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056448, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43e310, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21468K, capacity 21750K, committed 21960K, reserved 1069056K
  class space    used 2445K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810241115 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(195238760k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T11:15:34.052+0200: 1.648: [GC (Allocation Failure) 2018-10-24T11:15:34.052+0200: 1.648: [ParNew: 209792K->24380K(235968K), 0.0762337 secs] 209792K->40766K(2070976K), 0.0763600 secs] [Times: user=0.54 sys=0.02, real=0.07 secs] 
2018-10-24T11:16:36.116+0200: 63.712: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 80153K(2070976K), 0.0080867 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2018-10-24T11:16:36.125+0200: 63.721: [CMS-concurrent-mark-start]
2018-10-24T11:16:36.129+0200: 63.725: [CMS-concurrent-mark: 0.005/0.005 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2018-10-24T11:16:36.129+0200: 63.725: [CMS-concurrent-preclean-start]
2018-10-24T11:16:36.132+0200: 63.728: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
2018-10-24T11:16:36.132+0200: 63.728: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T11:16:41.173+0200: 68.769: [CMS-concurrent-abortable-preclean: 1.232/5.041 secs] [Times: user=1.23 sys=0.01, real=5.04 secs] 
2018-10-24T11:16:41.173+0200: 68.769: [GC (CMS Final Remark) [YG occupancy: 63767 K (235968 K)]2018-10-24T11:16:41.173+0200: 68.769: [Rescan (parallel) , 0.0077752 secs]2018-10-24T11:16:41.181+0200: 68.777: [weak refs processing, 0.0000230 secs]2018-10-24T11:16:41.181+0200: 68.777: [class unloading, 0.0029923 secs]2018-10-24T11:16:41.184+0200: 68.780: [scrub symbol table, 0.0040177 secs]2018-10-24T11:16:41.188+0200: 68.784: [scrub string table, 0.0003500 secs][1 CMS-remark: 16386K(1835008K)] 80153K(2070976K), 0.0158982 secs] [Times: user=0.07 sys=0.00, real=0.01 secs] 
2018-10-24T11:16:41.189+0200: 68.786: [CMS-concurrent-sweep-start]
2018-10-24T11:16:41.191+0200: 68.787: [CMS-concurrent-sweep: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
2018-10-24T11:16:41.191+0200: 68.787: [CMS-concurrent-reset-start]
2018-10-24T11:16:41.199+0200: 68.795: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
Heap
 par new generation   total 235968K, used 107999K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x00000000851a8c30, 0x000000008cce0000)
  from space 26176K,  93% used [0x000000008e670000, 0x000000008fe3f2d0, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 16386K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21555K, capacity 21808K, committed 22216K, reserved 1069056K
  class space    used 2411K, capacity 2488K, committed 2508K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291417 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184370952k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T14:17:37.703+0200: 1.261: [GC (Allocation Failure) 2018-10-29T14:17:37.704+0200: 1.262: [ParNew: 209792K->14139K(235968K), 0.0119574 secs] 209792K->14139K(2070976K), 0.0126222 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96404K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056618, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43ed88, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21425K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291456 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184356744k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T14:56:44.825+0200: 1.255: [GC (Allocation Failure) 2018-10-29T14:56:44.826+0200: 1.255: [ParNew: 209792K->14147K(235968K), 0.0134854 secs] 209792K->14147K(2070976K), 0.0142006 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96413K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056688, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f440e20, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21422K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291538 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184340624k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T15:38:34.008+0200: 1.302: [GC (Allocation Failure) 2018-10-29T15:38:34.009+0200: 1.302: [ParNew: 209792K->14151K(235968K), 0.0128067 secs] 209792K->14151K(2070976K), 0.0134940 secs] [Times: user=0.06 sys=0.00, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 98514K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262c10, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f441de8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21460K, capacity 21718K, committed 21960K, reserved 1069056K
  class space    used 2439K, capacity 2521K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810241138 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(194548592k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T11:38:02.324+0200: 1.282: [GC (Allocation Failure) 2018-10-24T11:38:02.324+0200: 1.282: [ParNew: 209792K->24379K(235968K), 0.1062387 secs] 209792K->40765K(2070976K), 0.1063616 secs] [Times: user=0.79 sys=0.01, real=0.10 secs] 
2018-10-24T11:39:04.417+0200: 63.375: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 78189K(2070976K), 0.0079674 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2018-10-24T11:39:04.425+0200: 63.383: [CMS-concurrent-mark-start]
2018-10-24T11:39:04.430+0200: 63.388: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2018-10-24T11:39:04.430+0200: 63.388: [CMS-concurrent-preclean-start]
2018-10-24T11:39:04.432+0200: 63.390: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
2018-10-24T11:39:04.432+0200: 63.390: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T11:39:09.526+0200: 68.484: [CMS-concurrent-abortable-preclean: 1.252/5.094 secs] [Times: user=1.25 sys=0.01, real=5.09 secs] 
2018-10-24T11:39:09.526+0200: 68.485: [GC (CMS Final Remark) [YG occupancy: 61803 K (235968 K)]2018-10-24T11:39:09.527+0200: 68.485: [Rescan (parallel) , 0.0073957 secs]2018-10-24T11:39:09.534+0200: 68.492: [weak refs processing, 0.0000243 secs]2018-10-24T11:39:09.534+0200: 68.492: [class unloading, 0.0031692 secs]2018-10-24T11:39:09.537+0200: 68.495: [scrub symbol table, 0.0045382 secs]2018-10-24T11:39:09.542+0200: 68.500: [scrub string table, 0.0004086 secs][1 CMS-remark: 16386K(1835008K)] 78189K(2070976K), 0.0161897 secs] [Times: user=0.07 sys=0.00, real=0.02 secs] 
2018-10-24T11:39:09.543+0200: 68.501: [CMS-concurrent-sweep-start]
2018-10-24T11:39:09.544+0200: 68.502: [CMS-concurrent-sweep: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T11:39:09.544+0200: 68.502: [CMS-concurrent-reset-start]
2018-10-24T11:39:09.552+0200: 68.510: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 
2018-10-24T13:29:03.258+0200: 6662.216: [GC (Allocation Failure) 2018-10-24T13:29:03.258+0200: 6662.216: [ParNew: 234171K->5658K(235968K), 0.0520074 secs] 250557K->37556K(2070976K), 0.0521378 secs] [Times: user=0.32 sys=0.02, real=0.05 secs] 
Heap
 par new generation   total 235968K, used 31930K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  12% used [0x0000000080000000, 0x00000000819a7c18, 0x000000008cce0000)
  from space 26176K,  21% used [0x000000008cce0000, 0x000000008d266bf8, 0x000000008e670000)
  to   space 26176K,   0% used [0x000000008e670000, 0x000000008e670000, 0x0000000090000000)
 concurrent mark-sweep generation total 1835008K, used 31897K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21912K, capacity 22192K, committed 22472K, reserved 1069056K
  class space    used 2411K, capacity 2488K, committed 2508K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291230 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184369936k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T12:30:17.616+0200: 1.293: [GC (Allocation Failure) 2018-10-29T12:30:17.617+0200: 1.294: [ParNew: 209792K->14145K(235968K), 0.0123403 secs] 209792K->14145K(2070976K), 0.0130008 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98508K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262e48, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f440540, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21423K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291303 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184368176k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:03:23.130+0200: 1.185: [GC (Allocation Failure) 2018-10-29T13:03:23.130+0200: 1.186: [ParNew: 209792K->14144K(235968K), 0.0145294 secs] 209792K->14144K(2070976K), 0.0146526 secs] [Times: user=0.06 sys=0.01, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 98508K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262df0, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f440228, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21416K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291319 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184368312k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:19:17.693+0200: 1.250: [GC (Allocation Failure) 2018-10-29T13:19:17.693+0200: 1.251: [ParNew: 209792K->14138K(235968K), 0.0131332 secs] 209792K->14138K(2070976K), 0.0137785 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98501K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262d80, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43ea70, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21425K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291339 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184376140k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:39:49.252+0200: 1.223: [GC (Allocation Failure) 2018-10-29T13:39:49.252+0200: 1.223: [ParNew: 209792K->14144K(235968K), 0.0143870 secs] 209792K->14144K(2070976K), 0.0151843 secs] [Times: user=0.05 sys=0.02, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96410K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x00000000850566c8, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f4403d0, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21422K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291536 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184172008k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T15:36:11.561+0200: 1.301: [GC (Allocation Failure) 2018-10-29T15:36:11.561+0200: 1.301: [ParNew: 209792K->14147K(235968K), 0.0145400 secs] 209792K->14147K(2070976K), 0.0147380 secs] [Times: user=0.06 sys=0.01, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 100608K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  41% used [0x0000000080000000, 0x000000008546f670, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f440c60, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21468K, capacity 21750K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810241341 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(187112732k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T13:41:25.777+0200: 1.302: [GC (Allocation Failure) 2018-10-24T13:41:25.777+0200: 1.302: [ParNew: 209792K->24580K(235968K), 0.1099037 secs] 209792K->40966K(2070976K), 0.1100410 secs] [Times: user=0.87 sys=0.02, real=0.11 secs] 
2018-10-24T13:42:27.873+0200: 63.398: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 84414K(2070976K), 0.0079132 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2018-10-24T13:42:27.881+0200: 63.406: [CMS-concurrent-mark-start]
2018-10-24T13:42:27.885+0200: 63.410: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2018-10-24T13:42:27.885+0200: 63.410: [CMS-concurrent-preclean-start]
2018-10-24T13:42:27.888+0200: 63.413: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T13:42:27.888+0200: 63.413: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T13:42:32.970+0200: 68.495: [CMS-concurrent-abortable-preclean: 1.270/5.082 secs] [Times: user=1.27 sys=0.01, real=5.08 secs] 
2018-10-24T13:42:32.970+0200: 68.495: [GC (CMS Final Remark) [YG occupancy: 68028 K (235968 K)]2018-10-24T13:42:32.970+0200: 68.495: [Rescan (parallel) , 0.0077338 secs]2018-10-24T13:42:32.978+0200: 68.503: [weak refs processing, 0.0000556 secs]2018-10-24T13:42:32.978+0200: 68.503: [class unloading, 0.0032177 secs]2018-10-24T13:42:32.981+0200: 68.506: [scrub symbol table, 0.0043939 secs]2018-10-24T13:42:32.986+0200: 68.510: [scrub string table, 0.0003463 secs][1 CMS-remark: 16386K(1835008K)] 84414K(2070976K), 0.0164618 secs] [Times: user=0.06 sys=0.00, real=0.02 secs] 
2018-10-24T13:42:32.987+0200: 68.512: [CMS-concurrent-sweep-start]
2018-10-24T13:42:32.988+0200: 68.512: [CMS-concurrent-sweep: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T13:42:32.988+0200: 68.512: [CMS-concurrent-reset-start]
2018-10-24T13:42:32.995+0200: 68.520: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 133373K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  51% used [0x0000000080000000, 0x0000000086a3e7a8, 0x000000008cce0000)
  from space 26176K,  93% used [0x000000008e670000, 0x000000008fe71020, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 16386K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21630K, capacity 21880K, committed 22140K, reserved 1069056K
  class space    used 2412K, capacity 2488K, committed 2508K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291213 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184468500k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T12:13:38.797+0200: 1.264: [GC (Allocation Failure) 2018-10-29T12:13:38.798+0200: 1.264: [ParNew: 209792K->14147K(235968K), 0.0127328 secs] 209792K->14147K(2070976K), 0.0133953 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96412K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056658, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f440d80, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21422K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-namenode-omiprihdp03ap.mufep.net.out.5 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201810291430 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184381788k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T14:30:38.013+0200: 1.275: [GC (Allocation Failure) 2018-10-29T14:30:38.014+0200: 1.275: [ParNew: 209792K->14162K(235968K), 0.0135859 secs] 209792K->14162K(2070976K), 0.0142507 secs] [Times: user=0.04 sys=0.02, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96429K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x00000000850569b0, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f444be8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21416K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291500 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184346728k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T15:00:37.691+0200: 1.251: [GC (Allocation Failure) 2018-10-29T15:00:37.692+0200: 1.252: [ParNew: 209792K->14133K(235968K), 0.0126972 secs] 209792K->14133K(2070976K), 0.0134547 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98497K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262e30, 0x000000008cce0000)
  from space 26176K,  53% used [0x000000008e670000, 0x000000008f43d6f8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21413K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291547 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184342928k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T15:47:22.279+0200: 1.240: [GC (Allocation Failure) 2018-10-29T15:47:22.280+0200: 1.240: [ParNew: 209792K->14147K(235968K), 0.0119965 secs] 209792K->14147K(2070976K), 0.0126317 secs] [Times: user=0.05 sys=0.01, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 98510K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262e28, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f440d30, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21461K, capacity 21750K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810241423 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(173589320k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T14:23:45.451+0200: 1.342: [GC (Allocation Failure) 2018-10-24T14:23:45.451+0200: 1.342: [ParNew: 209792K->24587K(235968K), 0.0642295 secs] 209792K->40973K(2070976K), 0.0643522 secs] [Times: user=0.52 sys=0.02, real=0.06 secs] 
2018-10-24T14:24:47.494+0200: 63.385: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 84334K(2070976K), 0.0082104 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2018-10-24T14:24:47.502+0200: 63.393: [CMS-concurrent-mark-start]
2018-10-24T14:24:47.505+0200: 63.396: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2018-10-24T14:24:47.505+0200: 63.396: [CMS-concurrent-preclean-start]
2018-10-24T14:24:47.508+0200: 63.399: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T14:24:47.508+0200: 63.399: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T14:24:52.564+0200: 68.455: [CMS-concurrent-abortable-preclean: 1.249/5.056 secs] [Times: user=1.25 sys=0.00, real=5.06 secs] 
2018-10-24T14:24:52.564+0200: 68.455: [GC (CMS Final Remark) [YG occupancy: 67948 K (235968 K)]2018-10-24T14:24:52.564+0200: 68.455: [Rescan (parallel) , 0.0076604 secs]2018-10-24T14:24:52.572+0200: 68.463: [weak refs processing, 0.0000248 secs]2018-10-24T14:24:52.572+0200: 68.463: [class unloading, 0.0028079 secs]2018-10-24T14:24:52.575+0200: 68.466: [scrub symbol table, 0.0036138 secs]2018-10-24T14:24:52.578+0200: 68.469: [scrub string table, 0.0003101 secs][1 CMS-remark: 16386K(1835008K)] 84334K(2070976K), 0.0150859 secs] [Times: user=0.07 sys=0.00, real=0.01 secs] 
2018-10-24T14:24:52.579+0200: 68.470: [CMS-concurrent-sweep-start]
2018-10-24T14:24:52.579+0200: 68.470: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T14:24:52.579+0200: 68.470: [CMS-concurrent-reset-start]
2018-10-24T14:24:52.588+0200: 68.479: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 117116K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  44% used [0x0000000080000000, 0x0000000085a5c380, 0x000000008cce0000)
  from space 26176K,  93% used [0x000000008e670000, 0x000000008fe72e58, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 16386K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21576K, capacity 21848K, committed 22140K, reserved 1069056K
  class space    used 2405K, capacity 2456K, committed 2508K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-omiprihdp03ap.mufep.net.out.5 <==
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-namenode-omiprihdp03ap.mufep.net.log <==
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
	Number of suppressed write-lock reports: 0
	Longest write-lock held interval: 0
2018-10-29 15:47:22,494 INFO  namenode.FSNamesystem (FSNamesystem.java:stopActiveServices(1302)) - Stopping services started for active state
2018-10-29 15:47:22,494 INFO  namenode.FSNamesystem (FSNamesystem.java:writeUnlock(1689)) - FSNamesystem write lock held for 0 ms via
java.lang.Thread.getStackTrace(Thread.java:1559)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:945)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1690)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1339)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.close(FSNamesystem.java:1760)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:918)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:716)
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:697)
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1001)
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1710)
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
	Number of suppressed write-lock reports: 0
	Longest write-lock held interval: 0
2018-10-29 15:47:22,494 INFO  namenode.FSNamesystem (FSNamesystem.java:stopStandbyServices(1392)) - Stopping services started for standby state
2018-10-29 15:47:22,495 ERROR namenode.NameNode (NameNode.java:main(1783)) - Failed to start namenode.
java.lang.IllegalStateException: Could not determine own NN ID in namespace 'OmiHdpPrdCluster'. Please ensure that this node is one of the machines listed as an NN RPC address, or configure dfs.ha.namenode.id
	at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
	at org.apache.hadoop.hdfs.HAUtil.getNameNodeIdOfOtherNode(HAUtil.java:164)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createBlockTokenSecretManager(BlockManager.java:442)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:334)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:781)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:716)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:697)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1001)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1710)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
2018-10-29 15:47:22,496 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2018-10-29 15:47:22,497 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at omiprihdp03ap.mufep.net/10.6.7.23
************************************************************/
==> /var/log/hadoop/hdfs/gc.log-201810291223 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184366120k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T12:23:08.681+0200: 1.264: [GC (Allocation Failure) 2018-10-29T12:23:08.681+0200: 1.264: [ParNew: 209792K->14139K(235968K), 0.0126677 secs] 209792K->14139K(2070976K), 0.0128856 secs] [Times: user=0.04 sys=0.01, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 100601K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  41% used [0x0000000080000000, 0x000000008546f5c8, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43ee78, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21418K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291325 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184371044k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:25:43.642+0200: 1.242: [GC (Allocation Failure) 2018-10-29T13:25:43.643+0200: 1.243: [ParNew: 209792K->14145K(235968K), 0.0125742 secs] 209792K->14145K(2070976K), 0.0133519 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96411K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056780, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f4405c8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21419K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291341 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184375248k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:41:15.377+0200: 1.251: [GC (Allocation Failure) 2018-10-29T13:41:15.377+0200: 1.251: [ParNew: 209792K->14131K(235968K), 0.0121842 secs] 209792K->14131K(2070976K), 0.0123911 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96396K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x00000000850564f0, 0x000000008cce0000)
  from space 26176K,  53% used [0x000000008e670000, 0x000000008f43cc20, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21424K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-namenode-omiprihdp03ap.mufep.net.out.1 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-zkfc-omiprihdp03ap.mufep.net.out.4 <==
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:136)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:187)
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201810241449 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(171667804k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T14:49:15.104+0200: 1.257: [GC (Allocation Failure) 2018-10-24T14:49:15.105+0200: 1.257: [ParNew: 209792K->24576K(235968K), 0.0926388 secs] 209792K->40962K(2070976K), 0.0927928 secs] [Times: user=0.74 sys=0.02, real=0.09 secs] 
2018-10-24T14:50:17.185+0200: 63.337: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 84159K(2070976K), 0.0077603 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2018-10-24T14:50:17.193+0200: 63.345: [CMS-concurrent-mark-start]
2018-10-24T14:50:17.197+0200: 63.349: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2018-10-24T14:50:17.197+0200: 63.349: [CMS-concurrent-preclean-start]
2018-10-24T14:50:17.199+0200: 63.352: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T14:50:17.199+0200: 63.352: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T14:50:22.275+0200: 68.427: [CMS-concurrent-abortable-preclean: 1.264/5.076 secs] [Times: user=1.26 sys=0.01, real=5.08 secs] 
2018-10-24T14:50:22.276+0200: 68.428: [GC (CMS Final Remark) [YG occupancy: 67773 K (235968 K)]2018-10-24T14:50:22.276+0200: 68.428: [Rescan (parallel) , 0.0076826 secs]2018-10-24T14:50:22.283+0200: 68.436: [weak refs processing, 0.0000249 secs]2018-10-24T14:50:22.283+0200: 68.436: [class unloading, 0.0028875 secs]2018-10-24T14:50:22.286+0200: 68.439: [scrub symbol table, 0.0038950 secs]2018-10-24T14:50:22.290+0200: 68.442: [scrub string table, 0.0003330 secs][1 CMS-remark: 16386K(1835008K)] 84159K(2070976K), 0.0156261 secs] [Times: user=0.07 sys=0.00, real=0.02 secs] 
2018-10-24T14:50:22.291+0200: 68.444: [CMS-concurrent-sweep-start]
2018-10-24T14:50:22.291+0200: 68.444: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T14:50:22.291+0200: 68.444: [CMS-concurrent-reset-start]
2018-10-24T14:50:22.300+0200: 68.452: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
Heap
 par new generation   total 235968K, used 199094K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  83% used [0x0000000080000000, 0x000000008aa6db80, 0x000000008cce0000)
  from space 26176K,  93% used [0x000000008e670000, 0x000000008fe70040, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 16386K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21750K, capacity 21976K, committed 22216K, reserved 1069056K
  class space    used 2405K, capacity 2456K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-namenode-omiprihdp03ap.mufep.net.out <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201810291323 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184371504k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:23:56.564+0200: 1.250: [GC (Allocation Failure) 2018-10-29T13:23:56.564+0200: 1.251: [ParNew: 209792K->14156K(235968K), 0.0121555 secs] 209792K->14156K(2070976K), 0.0122764 secs] [Times: user=0.06 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96421K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056608, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f443158, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21424K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291439 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184354416k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T14:39:57.276+0200: 1.240: [GC (Allocation Failure) 2018-10-29T14:39:57.277+0200: 1.241: [ParNew: 209792K->14131K(235968K), 0.0135101 secs] 209792K->14131K(2070976K), 0.0141651 secs] [Times: user=0.06 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96396K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x00000000850566d0, 0x000000008cce0000)
  from space 26176K,  53% used [0x000000008e670000, 0x000000008f43cca8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21421K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291507 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184351576k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T15:07:20.816+0200: 1.255: [GC (Allocation Failure) 2018-10-29T15:07:20.817+0200: 1.256: [ParNew: 209792K->14138K(235968K), 0.0126537 secs] 209792K->14138K(2070976K), 0.0133192 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98501K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262e38, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43e998, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21421K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-zkfc-omiprihdp03ap.mufep.net.log <==
STARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r 3091053c59a62c82d82c9f778c48bde5ef0a89a1; compiled by 'jenkins' on 2018-05-11T07:53Z
STARTUP_MSG:   java = 1.8.0_191
************************************************************/
2018-10-29 16:18:09,314 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(45)) - registered UNIX signal handlers for [TERM, HUP, INT]
2018-10-29 16:18:09,574 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(45)) - SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at omiprihdp03ap.mufep.net/10.6.7.23
************************************************************/
2018-10-29 16:22:02,483 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(45)) - STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG:   user = hdfs
STARTUP_MSG:   host = omiprihdp03ap.mufep.net/10.6.7.23
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.7.3.2.6.5.0-292
STARTUP_MSG:   classpath = /usr/hdp/2.6.5.0-292/hadoop/conf:/usr/hdp/2.6.5.0-292/hadoop/lib/ojdbc6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-hdfs-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-plugin-classloader-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-yarn-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/activation-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/joda-time-2.9.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsch-0.1.54.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/azure-storage-5.4.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/json-smart-1.3.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/junit-4.11.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-lang3-3.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/httpclient-4.5.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/httpcore-4.4.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.6.5.0-292/hadoop/.//azure-data-lake-store-sdk-2.2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/.//gcs-connector-1.8.1.2.6.5.0-292-shaded.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-annotations-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-annotations.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-auth-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-auth.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-aws-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-aws.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-azure-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-azure-datalake.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-azure.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-common-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-common-tests.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-common.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-nfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-nfs.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/./:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-core-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/netty-all-4.0.52.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/okhttp-2.7.5.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/okio-1.6.0.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jsch-0.1.54.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/azure-storage-5.4.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/json-smart-1.3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jsp-api-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-configuration-1.6.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-digester-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/metrics-core-3.0.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-lang3-3.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-net-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/objenesis-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/curator-framework-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/fst-2.24.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/gson-2.2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/httpclient-4.5.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/httpcore-4.4.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-core-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jets3t-0.9.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jcip-annotations-1.0-1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/zookeeper-3.4.6.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-api-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-client-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-registry-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//activation-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-rumen-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-gridmix-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-sls-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jcip-annotations-1.0-1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//azure-keyvault-core-0.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-sls.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jettison-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-digester-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//httpclient-4.5.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//httpcore-4.4.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-lang3-3.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//curator-framework-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-ant-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-ant.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//okhttp-2.7.5.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-archives-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-archives.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-auth-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jsch-0.1.54.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-auth.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-datajoin-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//json-smart-1.3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-distcp-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-openstack-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-extras-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//junit-4.11.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-extras.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//okio-1.6.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-streaming-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//zookeeper-3.4.6.2.6.5.0-292.jar::/usr/hdp/2.6.5.0-292/tez/tez-api-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-common-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-dag-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-examples-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-history-parser-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-job-analyzer-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-mapreduce-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-runtime-internals-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-runtime-library-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-tests-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/tez/lib/gcs-connector-1.8.1.2.6.5.0-292-shaded.jar:/usr/hdp/2.6.5.0-292/tez/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-annotations-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-aws-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-azure-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-yarn-server-web-proxy-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.6.5.0-292/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.6.5.0-292/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/2.6.5.0-292/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.6.5.0-292/tez/conf:/usr/hdp/2.6.5.0-292/tez/tez-api-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-common-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-dag-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-examples-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-history-parser-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-job-analyzer-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-mapreduce-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-runtime-internals-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-runtime-library-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-tests-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/tez/lib/gcs-connector-1.8.1.2.6.5.0-292-shaded.jar:/usr/hdp/2.6.5.0-292/tez/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-annotations-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-aws-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-azure-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-yarn-server-web-proxy-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.6.5.0-292/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.6.5.0-292/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/2.6.5.0-292/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.6.5.0-292/tez/conf
STARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r 3091053c59a62c82d82c9f778c48bde5ef0a89a1; compiled by 'jenkins' on 2018-05-11T07:53Z
STARTUP_MSG:   java = 1.8.0_191
************************************************************/
2018-10-29 16:22:02,493 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(45)) - registered UNIX signal handlers for [TERM, HUP, INT]
2018-10-29 16:22:02,724 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(45)) - SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at omiprihdp03ap.mufep.net/10.6.7.23
************************************************************/
2018-10-29 16:26:38,350 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(45)) - STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG:   user = hdfs
STARTUP_MSG:   host = omiprihdp03ap.mufep.net/10.6.7.23
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.7.3.2.6.5.0-292
STARTUP_MSG:   classpath = /usr/hdp/2.6.5.0-292/hadoop/conf:/usr/hdp/2.6.5.0-292/hadoop/lib/ojdbc6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-hdfs-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-plugin-classloader-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-yarn-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/activation-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/joda-time-2.9.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsch-0.1.54.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/azure-storage-5.4.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/json-smart-1.3.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/junit-4.11.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-lang3-3.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/httpclient-4.5.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/httpcore-4.4.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.6.5.0-292/hadoop/.//azure-data-lake-store-sdk-2.2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/.//gcs-connector-1.8.1.2.6.5.0-292-shaded.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-annotations-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-annotations.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-auth-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-auth.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-aws-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-aws.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-azure-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-azure-datalake.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-azure.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-common-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-common-tests.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-common.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-nfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/.//hadoop-nfs.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/./:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-core-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/netty-all-4.0.52.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/okhttp-2.7.5.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/okio-1.6.0.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jsch-0.1.54.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/azure-storage-5.4.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/json-smart-1.3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jsp-api-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-configuration-1.6.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-digester-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/metrics-core-3.0.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-lang3-3.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/commons-net-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/objenesis-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/curator-framework-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/fst-2.24.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/gson-2.2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/httpclient-4.5.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/httpcore-4.4.4.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-core-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jets3t-0.9.0.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jcip-annotations-1.0-1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/zookeeper-3.4.6.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-api-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-client-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-registry-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//activation-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-rumen-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-gridmix-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-sls-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jcip-annotations-1.0-1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//azure-keyvault-core-0.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-sls.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jettison-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-digester-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//httpclient-4.5.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//httpcore-4.4.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-lang3-3.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//curator-framework-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-ant-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-ant.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//okhttp-2.7.5.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-archives-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-archives.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-auth-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jsch-0.1.54.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-auth.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-datajoin-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//json-smart-1.3.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-distcp-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-openstack-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-extras-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//junit-4.11.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-extras.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//okio-1.6.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//hadoop-streaming-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//zookeeper-3.4.6.2.6.5.0-292.jar::/usr/hdp/2.6.5.0-292/tez/tez-api-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-common-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-dag-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-examples-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-history-parser-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-job-analyzer-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-mapreduce-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-runtime-internals-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-runtime-library-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-tests-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/tez/lib/gcs-connector-1.8.1.2.6.5.0-292-shaded.jar:/usr/hdp/2.6.5.0-292/tez/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-annotations-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-aws-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-azure-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-yarn-server-web-proxy-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.6.5.0-292/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.6.5.0-292/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/2.6.5.0-292/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.6.5.0-292/tez/conf:/usr/hdp/2.6.5.0-292/tez/tez-api-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-common-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-dag-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-examples-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-history-parser-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-job-analyzer-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-mapreduce-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-runtime-internals-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-runtime-library-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-tests-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/tez/lib/gcs-connector-1.8.1.2.6.5.0-292-shaded.jar:/usr/hdp/2.6.5.0-292/tez/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-annotations-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-aws-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-azure-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/hadoop-yarn-server-web-proxy-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.6.5.0-292/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.6.5.0-292/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.6.5.0-292/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/2.6.5.0-292/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.6.5.0-292/tez/conf
STARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r 3091053c59a62c82d82c9f778c48bde5ef0a89a1; compiled by 'jenkins' on 2018-05-11T07:53Z
STARTUP_MSG:   java = 1.8.0_191
************************************************************/
2018-10-29 16:26:38,360 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(45)) - registered UNIX signal handlers for [TERM, HUP, INT]
2018-10-29 16:26:38,585 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(45)) - SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at omiprihdp03ap.mufep.net/10.6.7.23
************************************************************/
==> /var/log/hadoop/hdfs/gc.log-201810241610 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(167802056k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T16:10:41.265+0200: 1.328: [GC (Allocation Failure) 2018-10-24T16:10:41.265+0200: 1.328: [ParNew: 209792K->24590K(235968K), 0.1116302 secs] 209792K->40976K(2070976K), 0.1117583 secs] [Times: user=0.88 sys=0.02, real=0.11 secs] 
2018-10-24T16:11:43.354+0200: 63.417: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 84338K(2070976K), 0.0077304 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2018-10-24T16:11:43.362+0200: 63.425: [CMS-concurrent-mark-start]
2018-10-24T16:11:43.366+0200: 63.429: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T16:11:43.366+0200: 63.429: [CMS-concurrent-preclean-start]
2018-10-24T16:11:43.368+0200: 63.431: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2018-10-24T16:11:43.368+0200: 63.431: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T16:11:48.437+0200: 68.500: [CMS-concurrent-abortable-preclean: 1.257/5.069 secs] [Times: user=1.26 sys=0.00, real=5.07 secs] 
2018-10-24T16:11:48.438+0200: 68.501: [GC (CMS Final Remark) [YG occupancy: 67952 K (235968 K)]2018-10-24T16:11:48.438+0200: 68.501: [Rescan (parallel) , 0.0075482 secs]2018-10-24T16:11:48.445+0200: 68.508: [weak refs processing, 0.0000230 secs]2018-10-24T16:11:48.445+0200: 68.508: [class unloading, 0.0031608 secs]2018-10-24T16:11:48.448+0200: 68.511: [scrub symbol table, 0.0040634 secs]2018-10-24T16:11:48.452+0200: 68.515: [scrub string table, 0.0003496 secs][1 CMS-remark: 16386K(1835008K)] 84338K(2070976K), 0.0157823 secs] [Times: user=0.06 sys=0.00, real=0.02 secs] 
2018-10-24T16:11:48.454+0200: 68.517: [CMS-concurrent-sweep-start]
2018-10-24T16:11:48.455+0200: 68.518: [CMS-concurrent-sweep: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T16:11:48.455+0200: 68.518: [CMS-concurrent-reset-start]
2018-10-24T16:11:48.463+0200: 68.526: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 145259K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  57% used [0x0000000080000000, 0x00000000875d7378, 0x000000008cce0000)
  from space 26176K,  93% used [0x000000008e670000, 0x000000008fe73bb0, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 16386K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21658K, capacity 21916K, committed 22140K, reserved 1069056K
  class space    used 2405K, capacity 2456K, committed 2508K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291249 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184383344k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T12:49:16.103+0200: 1.239: [GC (Allocation Failure) 2018-10-29T12:49:16.104+0200: 1.239: [ParNew: 209792K->14140K(235968K), 0.0116747 secs] 209792K->14140K(2070976K), 0.0123039 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98503K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262d80, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43f0a8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21420K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291304 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184374808k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:04:04.008+0200: 1.269: [GC (Allocation Failure) 2018-10-29T13:04:04.008+0200: 1.269: [ParNew: 209792K->14150K(235968K), 0.0130799 secs] 209792K->14150K(2070976K), 0.0132757 secs] [Times: user=0.06 sys=0.00, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 96416K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056940, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f4419d8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21423K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291406 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184367832k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T14:06:42.892+0200: 1.229: [GC (Allocation Failure) 2018-10-29T14:06:42.892+0200: 1.230: [ParNew: 209792K->14150K(235968K), 0.0118410 secs] 209792K->14150K(2070976K), 0.0124932 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98504K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085260788, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f441b80, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21418K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-namenode-omiprihdp03ap.mufep.net.out.2 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201810241655 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(167319644k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T16:55:52.756+0200: 1.334: [GC (Allocation Failure) 2018-10-24T16:55:52.756+0200: 1.334: [ParNew: 209792K->24951K(235968K), 0.1101571 secs] 209792K->41337K(2070976K), 0.1102825 secs] [Times: user=0.88 sys=0.02, real=0.11 secs] 
2018-10-24T16:56:54.843+0200: 63.421: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 88644K(2070976K), 0.0076252 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2018-10-24T16:56:54.851+0200: 63.429: [CMS-concurrent-mark-start]
2018-10-24T16:56:54.855+0200: 63.432: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T16:56:54.855+0200: 63.432: [CMS-concurrent-preclean-start]
2018-10-24T16:56:54.857+0200: 63.435: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2018-10-24T16:56:54.857+0200: 63.435: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T16:56:59.948+0200: 68.526: [CMS-concurrent-abortable-preclean: 1.382/5.091 secs] [Times: user=1.38 sys=0.00, real=5.09 secs] 
2018-10-24T16:56:59.948+0200: 68.526: [GC (CMS Final Remark) [YG occupancy: 72257 K (235968 K)]2018-10-24T16:56:59.948+0200: 68.526: [Rescan (parallel) , 0.0077662 secs]2018-10-24T16:56:59.956+0200: 68.534: [weak refs processing, 0.0000287 secs]2018-10-24T16:56:59.956+0200: 68.534: [class unloading, 0.0035862 secs]2018-10-24T16:56:59.960+0200: 68.538: [scrub symbol table, 0.0051221 secs]2018-10-24T16:56:59.965+0200: 68.543: [scrub string table, 0.0004291 secs][1 CMS-remark: 16386K(1835008K)] 88644K(2070976K), 0.0176956 secs] [Times: user=0.07 sys=0.00, real=0.02 secs] 
2018-10-24T16:56:59.966+0200: 68.544: [CMS-concurrent-sweep-start]
2018-10-24T16:56:59.967+0200: 68.545: [CMS-concurrent-sweep: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T16:56:59.967+0200: 68.545: [CMS-concurrent-reset-start]
2018-10-24T16:56:59.975+0200: 68.553: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 
2018-10-24T17:54:53.950+0200: 3542.527: [GC (Allocation Failure) 2018-10-24T17:54:53.950+0200: 3542.528: [ParNew: 234743K->7941K(235968K), 0.0690527 secs] 251129K->40213K(2070976K), 0.0691659 secs] [Times: user=0.44 sys=0.05, real=0.07 secs] 
2018-10-24T20:53:55.037+0200: 14283.615: [GC (Allocation Failure) 2018-10-24T20:53:55.037+0200: 14283.615: [ParNew: 217733K->2849K(235968K), 0.0067938 secs] 250005K->35121K(2070976K), 0.0068772 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96286K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  44% used [0x0000000080000000, 0x0000000085b3f620, 0x000000008cce0000)
  from space 26176K,  10% used [0x000000008e670000, 0x000000008e9384e8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 32272K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 26269K, capacity 26592K, committed 26952K, reserved 1073152K
  class space    used 2758K, capacity 2842K, committed 2944K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291444 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184357828k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T14:44:59.307+0200: 1.236: [GC (Allocation Failure) 2018-10-29T14:44:59.308+0200: 1.236: [ParNew: 209792K->14148K(235968K), 0.0120651 secs] 209792K->14148K(2070976K), 0.0127173 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98511K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262cd8, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f441280, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21424K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-namenode-omiprihdp03ap.mufep.net.out.4 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201810291313 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184374180k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:13:31.118+0200: 1.259: [GC (Allocation Failure) 2018-10-29T13:13:31.119+0200: 1.259: [ParNew: 209792K->14143K(235968K), 0.0125834 secs] 209792K->14143K(2070976K), 0.0127867 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 96409K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056878, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43fd18, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21423K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291440 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184353096k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T14:40:50.424+0200: 1.274: [GC (Allocation Failure) 2018-10-29T14:40:50.424+0200: 1.274: [ParNew: 209792K->14147K(235968K), 0.0122114 secs] 209792K->14147K(2070976K), 0.0124318 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98510K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262d40, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f440d28, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21420K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291521 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184339408k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T15:21:20.693+0200: 1.242: [GC (Allocation Failure) 2018-10-29T15:21:20.693+0200: 1.243: [ParNew: 209792K->14141K(235968K), 0.0123566 secs] 209792K->14141K(2070976K), 0.0132286 secs] [Times: user=0.04 sys=0.02, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 98504K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  40% used [0x0000000080000000, 0x0000000085262de8, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43f418, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21420K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810242222 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(165169792k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-24T22:22:18.570+0200: 1.304: [GC (Allocation Failure) 2018-10-24T22:22:18.570+0200: 1.304: [ParNew: 209792K->24934K(235968K), 0.0872132 secs] 209792K->41320K(2070976K), 0.0873404 secs] [Times: user=0.67 sys=0.02, real=0.09 secs] 
2018-10-24T22:23:20.636+0200: 63.370: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(1835008K)] 163007K(2070976K), 0.0111701 secs] [Times: user=0.07 sys=0.00, real=0.01 secs] 
2018-10-24T22:23:20.648+0200: 63.381: [CMS-concurrent-mark-start]
2018-10-24T22:23:20.651+0200: 63.385: [CMS-concurrent-mark: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2018-10-24T22:23:20.651+0200: 63.385: [CMS-concurrent-preclean-start]
2018-10-24T22:23:20.656+0200: 63.390: [CMS-concurrent-preclean: 0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T22:23:20.656+0200: 63.390: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2018-10-24T22:23:25.736+0200: 68.469: [CMS-concurrent-abortable-preclean: 1.325/5.079 secs] [Times: user=1.33 sys=0.00, real=5.08 secs] 
2018-10-24T22:23:25.736+0200: 68.470: [GC (CMS Final Remark) [YG occupancy: 147320 K (235968 K)]2018-10-24T22:23:25.736+0200: 68.470: [Rescan (parallel) , 0.0113383 secs]2018-10-24T22:23:25.747+0200: 68.481: [weak refs processing, 0.0000283 secs]2018-10-24T22:23:25.747+0200: 68.481: [class unloading, 0.0044830 secs]2018-10-24T22:23:25.752+0200: 68.486: [scrub symbol table, 0.0055488 secs]2018-10-24T22:23:25.757+0200: 68.491: [scrub string table, 0.0004413 secs][1 CMS-remark: 16386K(1835008K)] 163706K(2070976K), 0.0229673 secs] [Times: user=0.10 sys=0.00, real=0.02 secs] 
2018-10-24T22:23:25.759+0200: 68.493: [CMS-concurrent-sweep-start]
2018-10-24T22:23:25.760+0200: 68.493: [CMS-concurrent-sweep: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-24T22:23:25.760+0200: 68.493: [CMS-concurrent-reset-start]
2018-10-24T22:23:25.768+0200: 68.502: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 
2018-10-24T23:23:19.714+0200: 3662.447: [GC (Allocation Failure) 2018-10-24T23:23:19.714+0200: 3662.448: [ParNew: 234726K->7900K(235968K), 0.0647148 secs] 251112K->40176K(2070976K), 0.0648297 secs] [Times: user=0.38 sys=0.07, real=0.06 secs] 
2018-10-25T02:20:20.884+0200: 14283.617: [GC (Allocation Failure) 2018-10-25T02:20:20.884+0200: 14283.617: [ParNew: 217692K->2674K(235968K), 0.0091647 secs] 249968K->34949K(2070976K), 0.0092651 secs] [Times: user=0.06 sys=0.01, real=0.01 secs] 
2018-10-25T05:06:22.016+0200: 24244.750: [GC (Allocation Failure) 2018-10-25T05:06:22.016+0200: 24244.750: [ParNew: 212466K->2441K(235968K), 0.0073466 secs] 244741K->34717K(2070976K), 0.0074268 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2018-10-25T09:35:23.400+0200: 40386.134: [GC (Allocation Failure) 2018-10-25T09:35:23.400+0200: 40386.134: [ParNew: 212233K->2534K(235968K), 0.0068900 secs] 244509K->34809K(2070976K), 0.0069641 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 99362K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  46% used [0x0000000080000000, 0x0000000085e8f098, 0x000000008cce0000)
  from space 26176K,   9% used [0x000000008e670000, 0x000000008e8e99e8, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 32275K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 26578K, capacity 26848K, committed 27224K, reserved 1073152K
  class space    used 2754K, capacity 2842K, committed 2916K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-omiprihdp03ap.mufep.net.out.4 <==
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-omiprihdp03ap.mufep.net.out.3 <==
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-omiprihdp03ap.mufep.net.out.2 <==
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RetriableException): NameNode still not started
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkNNStartup(NameNodeRpcServer.java:2082)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getTransactionID(NameNodeRpcServer.java:1229)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.getTransactionId(NamenodeProtocolServerSideTranslatorPB.java:118)
	at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12832)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)

	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
	at org.apache.hadoop.ipc.Client.call(Client.java:1498)
	at org.apache.hadoop.ipc.Client.call(Client.java:1398)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
	at com.sun.proxy.$Proxy10.getTransactionId(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.getTransactionID(NamenodeProtocolTranslatorPB.java:130)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:290)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:202)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:184)
	at com.sun.proxy.$Proxy11.getTransactionID(Unknown Source)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.countUncheckpointedTxns(SecondaryNameNode.java:651)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.shouldCheckpointBasedOnCount(SecondaryNameNode.java:659)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:403)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:371)
	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:476)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:367)
	at java.lang.Thread.run(Thread.java:748)
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-omiprihdp03ap.mufep.net.out.1 <==
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-omiprihdp03ap.mufep.net.out <==
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:405)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:371)
	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:476)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:367)
	at java.lang.Thread.run(Thread.java:748)
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Log not rolled. Name node is in safe mode.
It was turned on manually. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1422)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:6309)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:1247)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:144)
	at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12836)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)

	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
	at org.apache.hadoop.ipc.Client.call(Client.java:1498)
	at org.apache.hadoop.ipc.Client.call(Client.java:1398)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
	at com.sun.proxy.$Proxy10.rollEditLog(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:290)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:202)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:184)
	at com.sun.proxy.$Proxy11.rollEditLog(Unknown Source)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:522)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:405)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:371)
	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:476)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:367)
	at java.lang.Thread.run(Thread.java:748)
==> /var/log/hadoop/hdfs/gc.log-201810251035 <==
2018-10-27T04:12:41.342+0200: 149836.581: [GC (Allocation Failure) 2018-10-27T04:12:41.342+0200: 149836.581: [ParNew: 210525K->605K(235968K), 0.0069253 secs] 244014K->34098K(2070976K), 0.0070102 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] 
2018-10-27T05:50:41.987+0200: 155717.225: [GC (Allocation Failure) 2018-10-27T05:50:41.987+0200: 155717.225: [ParNew: 210397K->665K(235968K), 0.0053941 secs] 243890K->34160K(2070976K), 0.0054826 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2018-10-27T07:36:42.535+0200: 162077.773: [GC (Allocation Failure) 2018-10-27T07:36:42.535+0200: 162077.773: [ParNew: 210457K->684K(235968K), 0.0060738 secs] 243952K->34179K(2070976K), 0.0061472 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] 
2018-10-27T09:38:42.526+0200: 169397.765: [GC (Allocation Failure) 2018-10-27T09:38:42.526+0200: 169397.765: [ParNew: 210476K->654K(235968K), 0.0062458 secs] 243971K->34179K(2070976K), 0.0063217 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2018-10-27T10:52:43.534+0200: 173838.772: [GC (Allocation Failure) 2018-10-27T10:52:43.534+0200: 173838.773: [ParNew: 210446K->449K(235968K), 0.0052087 secs] 243971K->34114K(2070976K), 0.0052878 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 
2018-10-27T12:52:44.148+0200: 181039.386: [GC (Allocation Failure) 2018-10-27T12:52:44.148+0200: 181039.386: [ParNew: 210241K->584K(235968K), 0.0064877 secs] 243906K->34249K(2070976K), 0.0065748 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2018-10-27T14:53:44.821+0200: 188300.060: [GC (Allocation Failure) 2018-10-27T14:53:44.822+0200: 188300.060: [ParNew: 210376K->765K(235968K), 0.0064116 secs] 244041K->34430K(2070976K), 0.0064888 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] 
2018-10-27T16:31:45.355+0200: 194180.593: [GC (Allocation Failure) 2018-10-27T16:31:45.355+0200: 194180.593: [ParNew: 210557K->594K(235968K), 0.0061077 secs] 244222K->34324K(2070976K), 0.0062066 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2018-10-27T18:07:45.924+0200: 199941.163: [GC (Allocation Failure) 2018-10-27T18:07:45.924+0200: 199941.163: [ParNew: 210386K->670K(235968K), 0.0047476 secs] 244116K->34409K(2070976K), 0.0048352 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 
2018-10-27T19:46:42.537+0200: 205877.775: [GC (Allocation Failure) 2018-10-27T19:46:42.537+0200: 205877.775: [ParNew: 210462K->707K(235968K), 0.0071229 secs] 244201K->34447K(2070976K), 0.0072222 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2018-10-27T21:24:42.521+0200: 211757.759: [GC (Allocation Failure) 2018-10-27T21:24:42.521+0200: 211757.759: [ParNew: 210499K->542K(235968K), 0.0057302 secs] 244239K->34349K(2070976K), 0.0058097 secs] [Times: user=0.02 sys=0.01, real=0.01 secs] 
2018-10-27T22:38:47.452+0200: 216202.690: [GC (Allocation Failure) 2018-10-27T22:38:47.452+0200: 216202.690: [ParNew: 210334K->617K(235968K), 0.0067979 secs] 244141K->34424K(2070976K), 0.0068865 secs] [Times: user=0.04 sys=0.01, real=0.00 secs] 
2018-10-28T00:39:42.520+0200: 223457.759: [GC (Allocation Failure) 2018-10-28T00:39:42.520+0200: 223457.759: [ParNew: 210409K->700K(235968K), 0.0068970 secs] 244216K->34507K(2070976K), 0.0069853 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2018-10-28T02:41:58.556+0200: 230793.794: [GC (Allocation Failure) 2018-10-28T02:41:58.556+0200: 230793.794: [ParNew: 210492K->755K(235968K), 0.0065499 secs] 244299K->34564K(2070976K), 0.0066532 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2018-10-28T04:36:49.325+0200: 237684.564: [GC (Allocation Failure) 2018-10-28T04:36:49.325+0200: 237684.564: [ParNew: 210547K->759K(235968K), 0.0048834 secs] 244356K->34634K(2070976K), 0.0049761 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 
2018-10-28T06:36:50.000+0200: 244885.238: [GC (Allocation Failure) 2018-10-28T06:36:50.000+0200: 244885.238: [ParNew: 210551K->713K(235968K), 0.0070871 secs] 244426K->34589K(2070976K), 0.0071777 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
2018-10-28T08:39:42.527+0200: 252257.765: [GC (Allocation Failure) 2018-10-28T08:39:42.527+0200: 252257.765: [ParNew: 210505K->751K(235968K), 0.0059436 secs] 244381K->34627K(2070976K), 0.0060241 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2018-10-28T10:36:51.347+0200: 259286.585: [GC (Allocation Failure) 2018-10-28T10:36:51.347+0200: 259286.585: [ParNew: 210543K->662K(235968K), 0.0059278 secs] 244419K->34601K(2070976K), 0.0060151 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2018-10-28T12:12:51.913+0200: 265047.151: [GC (Allocation Failure) 2018-10-28T12:12:51.913+0200: 265047.152: [ParNew: 210454K->648K(235968K), 0.0062106 secs] 244393K->34591K(2070976K), 0.0062920 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] 
2018-10-28T14:14:52.551+0200: 272367.789: [GC (Allocation Failure) 2018-10-28T14:14:52.551+0200: 272367.789: [ParNew: 210440K->773K(235968K), 0.0048926 secs] 244383K->34716K(2070976K), 0.0049833 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 
2018-10-28T15:52:57.529+0200: 278252.768: [GC (Allocation Failure) 2018-10-28T15:52:57.529+0200: 278252.768: [ParNew: 210565K->483K(235968K), 0.0061652 secs] 244508K->34573K(2070976K), 0.0063145 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2018-10-28T17:31:53.681+0200: 284188.920: [GC (Allocation Failure) 2018-10-28T17:31:53.682+0200: 284188.920: [ParNew: 210275K->535K(235968K), 0.0051891 secs] 244365K->34636K(2070976K), 0.0052850 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 
2018-10-28T19:09:54.206+0200: 290069.445: [GC (Allocation Failure) 2018-10-28T19:09:54.206+0200: 290069.445: [ParNew: 210327K->560K(235968K), 0.0045909 secs] 244428K->34662K(2070976K), 0.0046698 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2018-10-28T21:10:54.829+0200: 297330.067: [GC (Allocation Failure) 2018-10-28T21:10:54.829+0200: 297330.067: [ParNew: 210352K->500K(235968K), 0.0055804 secs] 244454K->34602K(2070976K), 0.0056861 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2018-10-28T22:36:55.350+0200: 302490.589: [GC (Allocation Failure) 2018-10-28T22:36:55.350+0200: 302490.589: [ParNew: 210292K->532K(235968K), 0.0048859 secs] 244394K->34675K(2070976K), 0.0049635 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2018-10-29T00:21:55.991+0200: 308791.229: [GC (Allocation Failure) 2018-10-29T00:21:55.991+0200: 308791.229: [ParNew: 210324K->589K(235968K), 0.0054767 secs] 244467K->34731K(2070976K), 0.0055682 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] 
2018-10-29T02:23:56.643+0200: 316111.881: [GC (Allocation Failure) 2018-10-29T02:23:56.643+0200: 316111.881: [ParNew: 210381K->585K(235968K), 0.0061674 secs] 244523K->34728K(2070976K), 0.0062612 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] 
2018-10-29T04:07:42.512+0200: 322337.750: [GC (Allocation Failure) 2018-10-29T04:07:42.512+0200: 322337.750: [ParNew: 210377K->503K(235968K), 0.0042246 secs] 244520K->34707K(2070976K), 0.0042889 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 
2018-10-29T05:27:57.787+0200: 327153.025: [GC (Allocation Failure) 2018-10-29T05:27:57.787+0200: 327153.025: [ParNew: 210295K->495K(235968K), 0.0061271 secs] 244499K->34700K(2070976K), 0.0062285 secs] [Times: user=0.03 sys=0.02, real=0.01 secs] 
2018-10-29T07:32:58.509+0200: 334653.748: [GC (Allocation Failure) 2018-10-29T07:32:58.509+0200: 334653.748: [ParNew: 210287K->625K(235968K), 0.0061059 secs] 244492K->34829K(2070976K), 0.0061912 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2018-10-29T09:09:59.034+0200: 340474.272: [GC (Allocation Failure) 2018-10-29T09:09:59.034+0200: 340474.272: [ParNew: 210417K->691K(235968K), 0.0064946 secs] 244621K->34917K(2070976K), 0.0066242 secs] [Times: user=0.04 sys=0.00, real=0.00 secs] 
2018-10-29T10:45:59.610+0200: 346234.849: [GC (Allocation Failure) 2018-10-29T10:45:59.610+0200: 346234.849: [ParNew: 210483K->598K(235968K), 0.0061810 secs] 244709K->34868K(2070976K), 0.0062636 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 145304K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  68% used [0x0000000080000000, 0x0000000088d50a40, 0x000000008cce0000)
  from space 26176K,   2% used [0x000000008cce0000, 0x000000008cd75858, 0x000000008e670000)
  to   space 26176K,   0% used [0x000000008e670000, 0x000000008e670000, 0x0000000090000000)
 concurrent mark-sweep generation total 1835008K, used 34270K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 26848K, capacity 27070K, committed 27404K, reserved 1073152K
  class space    used 2769K, capacity 2847K, committed 2864K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291212 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184492108k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
Heap
 par new generation   total 235968K, used 92325K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  44% used [0x0000000080000000, 0x0000000085a294a8, 0x000000008cce0000)
  from space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
  to   space 26176K,   0% used [0x000000008e670000, 0x000000008e670000, 0x0000000090000000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 10752K, capacity 10886K, committed 11008K, reserved 1058816K
  class space    used 1146K, capacity 1221K, committed 1280K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291216 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184465952k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T12:16:35.710+0200: 1.255: [GC (Allocation Failure) 2018-10-29T12:16:35.710+0200: 1.255: [ParNew: 209792K->14139K(235968K), 0.0133716 secs] 209792K->14139K(2070976K), 0.0140281 secs] [Times: user=0.04 sys=0.02, real=0.02 secs] 
Heap
 par new generation   total 235968K, used 96404K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x00000000850565c8, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43ed68, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21417K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2440K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201810291332 <==
OpenJDK 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct  9 2018 08:21:41 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 197551308k(184378432k free), swap 16777212k(16777212k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=268435456 -XX:MaxTenuringThreshold=6 -XX:NewSize=268435456 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-10-29T13:32:34.683+0200: 1.251: [GC (Allocation Failure) 2018-10-29T13:32:34.683+0200: 1.252: [ParNew: 209792K->14143K(235968K), 0.0138683 secs] 209792K->14143K(2070976K), 0.0145293 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 235968K, used 96408K [0x0000000080000000, 0x0000000090000000, 0x0000000090000000)
  eden space 209792K,  39% used [0x0000000080000000, 0x0000000085056510, 0x000000008cce0000)
  from space 26176K,  54% used [0x000000008e670000, 0x000000008f43fd30, 0x0000000090000000)
  to   space 26176K,   0% used [0x000000008cce0000, 0x000000008cce0000, 0x000000008e670000)
 concurrent mark-sweep generation total 1835008K, used 0K [0x0000000090000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21424K, capacity 21686K, committed 21960K, reserved 1069056K
  class space    used 2436K, capacity 2553K, committed 2560K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-zkfc-omiprihdp03ap.mufep.net.out.3 <==
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:136)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:187)
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-zkfc-omiprihdp03ap.mufep.net.out.2 <==
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:136)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:187)
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-zkfc-omiprihdp03ap.mufep.net.out.1 <==
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:136)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:187)
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-zkfc-omiprihdp03ap.mufep.net.out <==
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:136)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:187)
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-zkfc-omiprihdp03ap.mufep.net.out.5 <==
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:136)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:187)
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768541
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Command failed after 1 tries
3,049 Views
0 Kudos
1 REPLY 1

avatar
author-rank bjorn_olsen
Contributor

Created ‎10-30-2018 01:10 PM

Hi Sherrine

The ZKFC expects to run on the same host as the NN service. Assuming that this is what has been configured, the issue might be networking related.

Please check your hostname / IP configuration on your hosts, and that DNS is also giving the same correct result.

2,511 Views
0 Kudos
Announcements
Product Announcements
[ANNOUNCE] Cloudera on Public Cloud Runtime 7.2.18 is GA!
What's New @ Cloudera
Cloudera Operational Database (COD) supports enhancements to...
Community Announcements
March 2024 Community Highlights
Community Announcements
Celebrating 100,000 Members: A Journey of Growth and Connect...
Product Announcements
[ANNOUNCE] New Cloudera JDBC Connector for Apache Hive 2.6.2...
View More Announcements