stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 155, in <module>
    DataNode().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 978, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 62, in start
    datanode(action="start")
  File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py", line 68, in datanode
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 276, in service
    Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/2.6.5.0-292/hadoop/sbin/hadoop-daemon.sh --config /usr/hdp/2.6.5.0-292/hadoop/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev-hdfs-restore-clstr-w1.out
Error opening zip file or JAR manifest missing : /home/sshhdfsuser/jmx_prometheus_javaagent-0.11.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument
 stdout:
2019-06-06 10:36:52,460 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.0-292 -> 2.6.5.0-292
2019-06-06 10:36:52,478 - Using hadoop conf dir: /usr/hdp/2.6.5.0-292/hadoop/conf
2019-06-06 10:36:52,662 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.0-292 -> 2.6.5.0-292
2019-06-06 10:36:52,668 - Using hadoop conf dir: /usr/hdp/2.6.5.0-292/hadoop/conf
2019-06-06 10:36:52,669 - Group['hdfs'] {}
2019-06-06 10:36:52,670 - Group['hadoop'] {}
2019-06-06 10:36:52,671 - Group['users'] {}
2019-06-06 10:36:52,671 - Group['knox'] {}
2019-06-06 10:36:52,671 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2019-06-06 10:36:52,673 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2019-06-06 10:36:52,674 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2019-06-06 10:36:52,675 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2019-06-06 10:36:52,676 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs'], 'uid': None}
2019-06-06 10:36:52,677 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2019-06-06 10:36:52,678 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2019-06-06 10:36:52,679 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2019-06-06 10:36:52,680 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-06-06 10:36:52,681 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2019-06-06 10:36:52,688 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2019-06-06 10:36:52,689 - Group['hdfs'] {}
2019-06-06 10:36:52,690 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', u'hdfs']}
2019-06-06 10:36:52,690 - FS Type: 
2019-06-06 10:36:52,690 - Directory['/etc/hadoop'] {'mode': 0755}
2019-06-06 10:36:52,707 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-06-06 10:36:52,708 - Writing File['/usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-env.sh'] because contents don't match
2019-06-06 10:36:52,708 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2019-06-06 10:36:52,726 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2019-06-06 10:36:52,734 - Skipping Execute[('setenforce', '0')] due to not_if
2019-06-06 10:36:52,734 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2019-06-06 10:36:52,737 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2019-06-06 10:36:52,737 - Changing owner for /var/run/hadoop from 988 to root
2019-06-06 10:36:52,737 - Changing group for /var/run/hadoop from 987 to root
2019-06-06 10:36:52,738 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2019-06-06 10:36:52,742 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2019-06-06 10:36:52,744 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2019-06-06 10:36:52,751 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2019-06-06 10:36:52,761 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-06-06 10:36:52,761 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2019-06-06 10:36:52,762 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2019-06-06 10:36:52,767 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2019-06-06 10:36:52,771 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2019-06-06 10:36:53,161 - Using hadoop conf dir: /usr/hdp/2.6.5.0-292/hadoop/conf
2019-06-06 10:36:53,163 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.0-292 -> 2.6.5.0-292
2019-06-06 10:36:53,185 - Using hadoop conf dir: /usr/hdp/2.6.5.0-292/hadoop/conf
2019-06-06 10:36:53,203 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/2.6.5.0-292/hadoop/sbin/hadoop-daemon.sh --config /usr/hdp/2.6.5.0-292/hadoop/conf stop datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/2.6.5.0-292/hadoop/libexec'}, 'only_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2019-06-06 10:36:58,332 - call['! ( ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid )'] {'tries': 6, 'try_sleep': 10, 'env': {'PATH': '/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent', 'HADOOP_LIBEXEC_DIR': '/usr/hdp/2.6.5.0-292/hadoop/libexec'}}
2019-06-06 10:36:58,338 - call returned (0, '')
2019-06-06 10:36:58,339 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete']}
2019-06-06 10:36:58,340 - checked_call['/usr/hdp/current/hadoop-hdfs-datanode/bin/hdfs dfsadmin -fs hdfs://dev-hdfs-restore-clstr-m0.0tokzaoqb3aehp1kdyxu4rf1og.bx.internal.cloudapp.net:8020 -D ipc.client.connect.max.retries=5 -D ipc.client.connect.retry.interval=1000 -getDatanodeInfo 0.0.0.0:8010'] {'tries': 1, 'user': 'hdfs'}
2019-06-06 10:37:04,664 - DataNode has successfully shutdown.
2019-06-06 10:37:04,665 - Pid file /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid is empty or does not exist
2019-06-06 10:37:04,667 - Trying to acquire a lock on /var/lib/ambari-agent/tmp/link_configs_lock_file
2019-06-06 10:37:04,667 - Acquired the lock on /var/lib/ambari-agent/tmp/link_configs_lock_file
2019-06-06 10:37:04,669 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2019-06-06 10:37:04,674 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2019-06-06 10:37:04,675 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'configuration_attributes': {}, 'configurations': ...}
2019-06-06 10:37:04,685 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-policy.xml
2019-06-06 10:37:04,685 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-06-06 10:37:04,695 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'configuration_attributes': {}, 'configurations': ...}
2019-06-06 10:37:04,703 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/ssl-client.xml
2019-06-06 10:37:04,703 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-06-06 10:37:04,710 - Directory['/usr/hdp/2.6.5.0-292/hadoop/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2019-06-06 10:37:04,711 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2019-06-06 10:37:04,719 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/secure/ssl-client.xml
2019-06-06 10:37:04,719 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-06-06 10:37:04,725 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'configuration_attributes': {}, 'configurations': ...}
2019-06-06 10:37:04,733 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/ssl-server.xml
2019-06-06 10:37:04,733 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-06-06 10:37:04,741 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'configurations': ...}
2019-06-06 10:37:04,749 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/hdfs-site.xml
2019-06-06 10:37:04,750 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-06-06 10:37:04,799 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/2.6.5.0-292/hadoop/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2019-06-06 10:37:04,807 - Generating config: /usr/hdp/2.6.5.0-292/hadoop/conf/core-site.xml
2019-06-06 10:37:04,808 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2019-06-06 10:37:04,850 - File['/usr/hdp/2.6.5.0-292/hadoop/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'}
2019-06-06 10:37:04,850 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.0-292 -> 2.6.5.0-292
2019-06-06 10:37:04,854 - Directory['/var/lib/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'group': 'hadoop', 'mode': 0751}
2019-06-06 10:37:04,854 - Directory['/var/lib/ambari-agent/data/datanode'] {'create_parents': True, 'mode': 0755}
2019-06-06 10:37:04,861 - Host contains mounts: ['/sys', '/proc', '/dev', '/sys/kernel/security', '/dev/shm', '/dev/pts', '/run', '/sys/fs/cgroup', '/sys/fs/cgroup/systemd', '/sys/fs/pstore', '/sys/fs/cgroup/blkio', '/sys/fs/cgroup/cpu,cpuacct', '/sys/fs/cgroup/cpuset', '/sys/fs/cgroup/hugetlb', '/sys/fs/cgroup/freezer', '/sys/fs/cgroup/net_cls,net_prio', '/sys/fs/cgroup/pids', '/sys/fs/cgroup/devices', '/sys/fs/cgroup/memory', '/sys/fs/cgroup/perf_event', '/sys/kernel/config', '/', '/proc/sys/fs/binfmt_misc', '/sys/kernel/debug', '/dev/hugepages', '/dev/mqueue', '/hadoopfs/fs1', '/boot', '/mnt/resource', '/run/user/1007', '/run/user/989', '/run/user/1011'].
2019-06-06 10:37:04,862 - Mount point for directory /hadoopfs/fs1/hdfs/datanode is /hadoopfs/fs1
2019-06-06 10:37:04,862 - Mount point for directory /hadoopfs/fs1/hdfs/datanode is /hadoopfs/fs1
2019-06-06 10:37:04,862 - Forcefully ensuring existence and permissions of the directory: /hadoopfs/fs1/hdfs/datanode
2019-06-06 10:37:04,863 - Directory['/hadoopfs/fs1/hdfs/datanode'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'ignore_failures': True, 'mode': 0750, 'owner': 'hdfs'}
2019-06-06 10:37:04,869 - Host contains mounts: ['/sys', '/proc', '/dev', '/sys/kernel/security', '/dev/shm', '/dev/pts', '/run', '/sys/fs/cgroup', '/sys/fs/cgroup/systemd', '/sys/fs/pstore', '/sys/fs/cgroup/blkio', '/sys/fs/cgroup/cpu,cpuacct', '/sys/fs/cgroup/cpuset', '/sys/fs/cgroup/hugetlb', '/sys/fs/cgroup/freezer', '/sys/fs/cgroup/net_cls,net_prio', '/sys/fs/cgroup/pids', '/sys/fs/cgroup/devices', '/sys/fs/cgroup/memory', '/sys/fs/cgroup/perf_event', '/sys/kernel/config', '/', '/proc/sys/fs/binfmt_misc', '/sys/kernel/debug', '/dev/hugepages', '/dev/mqueue', '/hadoopfs/fs1', '/boot', '/mnt/resource', '/run/user/1007', '/run/user/989', '/run/user/1011'].
2019-06-06 10:37:04,869 - Mount point for directory /hadoopfs/fs1/hdfs/datanode is /hadoopfs/fs1
2019-06-06 10:37:04,869 - File['/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist'] {'content': '\n# This file keeps track of the last known mount-point for each dir.\n# It is safe to delete, since it will get regenerated the next time that the component of the service starts.\n# However, it is not advised to delete this file since Ambari may\n# re-create a dir that used to be mounted on a drive but is now mounted on the root.\n# Comments begin with a hash (#) symbol\n# dir,mount_point\n/hadoopfs/fs1/hdfs/datanode,/hadoopfs/fs1\n', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2019-06-06 10:37:04,870 - Releasing the lock on /var/lib/ambari-agent/tmp/link_configs_lock_file
2019-06-06 10:37:04,872 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2019-06-06 10:37:04,872 - Changing owner for /var/run/hadoop from 0 to hdfs
2019-06-06 10:37:04,872 - Changing group for /var/run/hadoop from 0 to hadoop
2019-06-06 10:37:04,873 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2019-06-06 10:37:04,873 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2019-06-06 10:37:04,874 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2019-06-06 10:37:04,879 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/2.6.5.0-292/hadoop/sbin/hadoop-daemon.sh --config /usr/hdp/2.6.5.0-292/hadoop/conf start datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/2.6.5.0-292/hadoop/libexec'}, 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2019-06-06 10:37:09,030 - Execute['find /var/log/hadoop/hdfs -maxdepth 1 -type f -name '*' -exec echo '==> {} <==' \; -exec tail -n 40 {} \;'] {'logoutput': True, 'ignore_failures': True, 'user': 'hdfs'}
==> /var/log/hadoop/hdfs/gc.log-201905231906 <==
2019-05-27T18:40:24.249+0000: 344054.661: [GC (Allocation Failure) 2019-05-27T18:40:24.249+0000: 344054.661: [ParNew: 164469K->748K(184320K), 0.0035584 secs] 176256K->12535K(1028096K), 0.0036328 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-27T19:25:17.756+0000: 346748.169: [GC (Allocation Failure) 2019-05-27T19:25:17.756+0000: 346748.169: [ParNew: 164588K->599K(184320K), 0.0066359 secs] 176375K->12386K(1028096K), 0.0067500 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-27T20:10:26.544+0000: 349456.956: [GC (Allocation Failure) 2019-05-27T20:10:26.544+0000: 349456.956: [ParNew: 164439K->563K(184320K), 0.0043787 secs] 176226K->12350K(1028096K), 0.0045339 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-27T20:56:34.246+0000: 352224.658: [GC (Allocation Failure) 2019-05-27T20:56:34.246+0000: 352224.658: [ParNew: 164403K->608K(184320K), 0.0041857 secs] 176190K->12395K(1028096K), 0.0044336 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-27T21:41:39.250+0000: 354929.662: [GC (Allocation Failure) 2019-05-27T21:41:39.250+0000: 354929.662: [ParNew: 164448K->621K(184320K), 0.0037129 secs] 176235K->12408K(1028096K), 0.0037959 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-27T22:26:15.498+0000: 357605.911: [GC (Allocation Failure) 2019-05-27T22:26:15.499+0000: 357605.911: [ParNew: 164461K->571K(184320K), 0.0039909 secs] 176248K->12358K(1028096K), 0.0041383 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-27T23:10:39.249+0000: 360269.661: [GC (Allocation Failure) 2019-05-27T23:10:39.249+0000: 360269.662: [ParNew: 164411K->540K(184320K), 0.0039222 secs] 176198K->12327K(1028096K), 0.0040125 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-27T23:57:06.597+0000: 363057.009: [GC (Allocation Failure) 2019-05-27T23:57:06.597+0000: 363057.009: [ParNew: 164380K->521K(184320K), 0.0040923 secs] 176167K->12308K(1028096K), 0.0041931 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-05-28T00:42:33.778+0000: 365784.190: [GC (Allocation Failure) 2019-05-28T00:42:33.778+0000: 365784.190: [ParNew: 164361K->565K(184320K), 0.0042841 secs] 176148K->12352K(1028096K), 0.0044941 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-28T01:27:49.251+0000: 368499.663: [GC (Allocation Failure) 2019-05-28T01:27:49.251+0000: 368499.663: [ParNew: 164405K->615K(184320K), 0.0035015 secs] 176192K->12402K(1028096K), 0.0036024 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T02:12:06.193+0000: 371156.605: [GC (Allocation Failure) 2019-05-28T02:12:06.193+0000: 371156.605: [ParNew: 164455K->558K(184320K), 0.0039758 secs] 176242K->12346K(1028096K), 0.0040654 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-05-28T02:57:14.247+0000: 373864.659: [GC (Allocation Failure) 2019-05-28T02:57:14.247+0000: 373864.659: [ParNew: 164398K->610K(184320K), 0.0040702 secs] 176186K->12397K(1028096K), 0.0041570 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-28T03:42:24.811+0000: 376575.223: [GC (Allocation Failure) 2019-05-28T03:42:24.811+0000: 376575.223: [ParNew: 164450K->639K(184320K), 0.0040451 secs] 176237K->12427K(1028096K), 0.0041280 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T04:27:50.597+0000: 379301.009: [GC (Allocation Failure) 2019-05-28T04:27:50.597+0000: 379301.009: [ParNew: 164479K->603K(184320K), 0.0043164 secs] 176267K->12391K(1028096K), 0.0044331 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-28T05:12:23.958+0000: 381974.370: [GC (Allocation Failure) 2019-05-28T05:12:23.958+0000: 381974.370: [ParNew: 164443K->637K(184320K), 0.0043379 secs] 176231K->12425K(1028096K), 0.0044261 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-28T05:57:43.485+0000: 384693.897: [GC (Allocation Failure) 2019-05-28T05:57:43.485+0000: 384693.898: [ParNew: 164477K->587K(184320K), 0.0036789 secs] 176265K->12374K(1028096K), 0.0038023 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-28T06:42:17.685+0000: 387368.098: [GC (Allocation Failure) 2019-05-28T06:42:17.686+0000: 387368.098: [ParNew: 164427K->593K(184320K), 0.0038606 secs] 176214K->12381K(1028096K), 0.0039445 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-28T07:27:04.249+0000: 390054.661: [GC (Allocation Failure) 2019-05-28T07:27:04.249+0000: 390054.661: [ParNew: 164433K->621K(184320K), 0.0040346 secs] 176221K->12409K(1028096K), 0.0041147 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T08:11:29.249+0000: 392719.661: [GC (Allocation Failure) 2019-05-28T08:11:29.249+0000: 392719.661: [ParNew: 164461K->589K(184320K), 0.0034062 secs] 176249K->12377K(1028096K), 0.0034841 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T08:55:40.911+0000: 395371.323: [GC (Allocation Failure) 2019-05-28T08:55:40.911+0000: 395371.323: [ParNew: 164429K->594K(184320K), 0.0038099 secs] 176217K->12382K(1028096K), 0.0038995 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T09:41:54.247+0000: 398144.659: [GC (Allocation Failure) 2019-05-28T09:41:54.247+0000: 398144.659: [ParNew: 164434K->678K(184320K), 0.0037659 secs] 176222K->12466K(1028096K), 0.0038945 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-28T10:27:15.540+0000: 400865.952: [GC (Allocation Failure) 2019-05-28T10:27:15.540+0000: 400865.952: [ParNew: 164518K->635K(184320K), 0.0034754 secs] 176306K->12423K(1028096K), 0.0036138 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T11:12:26.243+0000: 403576.655: [GC (Allocation Failure) 2019-05-28T11:12:26.243+0000: 403576.655: [ParNew: 164475K->569K(184320K), 0.0039078 secs] 176263K->12357K(1028096K), 0.0039904 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-05-28T11:57:26.378+0000: 406276.790: [GC (Allocation Failure) 2019-05-28T11:57:26.378+0000: 406276.790: [ParNew: 164409K->554K(184320K), 0.0038134 secs] 176197K->12342K(1028096K), 0.0038903 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-05-28T12:42:44.253+0000: 408994.665: [GC (Allocation Failure) 2019-05-28T12:42:44.253+0000: 408994.665: [ParNew: 164394K->642K(184320K), 0.0040208 secs] 176182K->12430K(1028096K), 0.0041165 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-05-28T13:27:37.231+0000: 411687.644: [GC (Allocation Failure) 2019-05-28T13:27:37.232+0000: 411687.644: [ParNew: 164482K->629K(184320K), 0.0037336 secs] 176270K->12417K(1028096K), 0.0038667 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T14:11:50.164+0000: 414340.576: [GC (Allocation Failure) 2019-05-28T14:11:50.164+0000: 414340.576: [ParNew: 164469K->667K(184320K), 0.0034653 secs] 176257K->12455K(1028096K), 0.0035487 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T14:56:38.822+0000: 417029.234: [GC (Allocation Failure) 2019-05-28T14:56:38.822+0000: 417029.234: [ParNew: 164507K->615K(184320K), 0.0041101 secs] 176295K->12403K(1028096K), 0.0042003 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-05-28T15:41:19.250+0000: 419709.662: [GC (Allocation Failure) 2019-05-28T15:41:19.250+0000: 419709.662: [ParNew: 164455K->675K(184320K), 0.0033534 secs] 176243K->12464K(1028096K), 0.0034550 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T16:27:14.249+0000: 422464.661: [GC (Allocation Failure) 2019-05-28T16:27:14.249+0000: 422464.661: [ParNew: 164515K->663K(184320K), 0.0037142 secs] 176304K->12452K(1028096K), 0.0039168 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-05-28T17:12:24.843+0000: 425175.255: [GC (Allocation Failure) 2019-05-28T17:12:24.843+0000: 425175.255: [ParNew: 164503K->582K(184320K), 0.0040731 secs] 176292K->12371K(1028096K), 0.0042497 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-05-28T17:59:39.209+0000: 428009.621: [GC (Allocation Failure) 2019-05-28T17:59:39.209+0000: 428009.621: [ParNew: 164422K->518K(184320K), 0.0033872 secs] 176211K->12307K(1028096K), 0.0034487 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 56588K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  34% used [0x00000000c0000000, 0x00000000c36c1818, 0x00000000ca000000)
  from space 20480K,   2% used [0x00000000cb400000, 0x00000000cb481978, 0x00000000cc800000)
  to   space 20480K,   0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
 concurrent mark-sweep generation total 843776K, used 11788K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 42840K, capacity 43280K, committed 43688K, reserved 1087488K
  class space    used 4579K, capacity 4712K, committed 4808K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev-hdfs-restore-clstr-w1.log <==
	... 11 more
2019-06-06 10:19:10,338 INFO  checker.ThrottledAsyncChecker (ThrottledAsyncChecker.java:schedule(122)) - Scheduling a check for /hadoopfs/fs1/hdfs/datanode/current
2019-06-06 10:19:10,338 WARN  datanode.DataNode (DataNode.java:run(2384)) - DatanodeRegistration(10.201.149.51:50010, datanodeUuid=ce57e7f1-c1b8-4167-8800-ea37c342d588, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-af1e3750-6e99-4024-ab86-3aa9e2ec0a1d;nsid=250023898;c=0):Failed to transfer BP-588423591-10.201.149.45-1558638648230:blk_1073742549_1725 to 10.201.149.50:50010 got 
java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
	at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
	at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:263)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:583)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:767)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:714)
	at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2352)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Connection reset by peer
	... 11 more
2019-06-06 10:33:48,068 INFO  web.DatanodeHttpServer (SimpleHttpProxyHandler.java:exceptionCaught(147)) - Proxy for / failed. cause: 
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:368)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:891)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:243)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)
2019-06-06 10:36:53,326 ERROR datanode.DataNode (LogAdapter.java:error(71)) - RECEIVED SIGNAL 15: SIGTERM
2019-06-06 10:36:53,329 INFO  datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at dev-hdfs-restore-clstr-w1.0tokzaoqb3aehp1kdyxu4rf1og.bx.internal.cloudapp.net/10.201.149.51
************************************************************/
==> /var/log/hadoop/hdfs/SecurityAuth.audit <==
2019-05-23 19:12:41,745 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:12:41,827 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:12:56,748 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:12:56,777 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:13:26,742 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:13:41,727 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:13:41,741 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:13:56,728 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:14:11,728 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:14:11,755 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:14:26,778 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:14:41,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:14:56,728 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:14:56,742 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:15:11,728 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:15:26,737 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:15:41,726 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:15:56,757 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:16:26,726 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:16:41,740 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:16:56,727 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:17:11,727 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:17:41,739 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:17:56,726 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:17:56,740 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:18:11,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2019-05-23 19:18:11,742 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
==> /var/log/hadoop/hdfs/hdfs-audit.log <==
==> /var/log/hadoop/hdfs/gc.log-201905281818 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(13357316k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-05-28T18:18:12.263+0000: 1.606: [GC (Allocation Failure) 2019-05-28T18:18:12.264+0000: 1.606: [ParNew: 163840K->13161K(184320K), 0.0235188 secs] 163840K->13161K(1028096K), 0.0236126 secs] [Times: user=0.05 sys=0.01, real=0.02 secs] 
2019-05-28T18:18:14.287+0000: 3.630: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 134299K(1028096K), 0.0130990 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
2019-05-28T18:18:14.301+0000: 3.643: [CMS-concurrent-mark-start]
2019-05-28T18:18:14.309+0000: 3.651: [CMS-concurrent-mark: 0.008/0.008 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-05-28T18:18:14.309+0000: 3.651: [CMS-concurrent-preclean-start]
2019-05-28T18:18:14.310+0000: 3.652: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-05-28T18:18:14.310+0000: 3.653: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-05-28T18:18:19.333+0000: 8.675: [CMS-concurrent-abortable-preclean: 1.237/5.023 secs] [Times: user=2.06 sys=0.03, real=5.03 secs] 
2019-05-28T18:18:19.333+0000: 8.675: [GC (CMS Final Remark) [YG occupancy: 169957 K (184320 K)]2019-05-28T18:18:19.333+0000: 8.675: [Rescan (parallel) , 0.0107270 secs]2019-05-28T18:18:19.344+0000: 8.686: [weak refs processing, 0.0000180 secs]2019-05-28T18:18:19.344+0000: 8.686: [class unloading, 0.0032024 secs]2019-05-28T18:18:19.347+0000: 8.689: [scrub symbol table, 0.0038104 secs]2019-05-28T18:18:19.351+0000: 8.693: [scrub string table, 0.0003833 secs][1 CMS-remark: 0K(843776K)] 169957K(1028096K), 0.0185523 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 
2019-05-28T18:18:19.352+0000: 8.694: [CMS-concurrent-sweep-start]
2019-05-28T18:18:19.352+0000: 8.694: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-05-28T18:18:19.352+0000: 8.694: [CMS-concurrent-reset-start]
2019-05-28T18:18:19.366+0000: 8.708: [CMS-concurrent-reset: 0.014/0.014 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 
2019-05-28T18:19:13.219+0000: 62.562: [GC (Allocation Failure) 2019-05-28T18:19:13.220+0000: 62.562: [ParNew: 177001K->12404K(184320K), 0.0647394 secs] 177001K->16655K(1028096K), 0.0648164 secs] [Times: user=0.08 sys=0.01, real=0.07 secs] 
2019-05-28T18:39:14.404+0000: 1263.746: [GC (Allocation Failure) 2019-05-28T18:39:14.404+0000: 1263.746: [ParNew: 176244K->7317K(184320K), 0.0109127 secs] 180495K->15708K(1028096K), 0.0110222 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 41028K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  20% used [0x00000000c0000000, 0x00000000c20ebca0, 0x00000000ca000000)
  from space 20480K,  35% used [0x00000000cb400000, 0x00000000cbb25558, 0x00000000cc800000)
  to   space 20480K,   0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
 concurrent mark-sweep generation total 843776K, used 8391K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 33182K, capacity 33590K, committed 34004K, reserved 1079296K
  class space    used 3784K, capacity 3895K, committed 3912K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev-hdfs-restore-clstr-w1.out <==
Error opening zip file or JAR manifest missing : /home/sshhdfsuser/jmx_prometheus_javaagent-0.11.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 55973
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201905281844 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12993464k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-05-28T18:44:14.002+0000: 1.583: [GC (Allocation Failure) 2019-05-28T18:44:14.002+0000: 1.583: [ParNew: 163840K->13665K(184320K), 0.0187156 secs] 163840K->13665K(1028096K), 0.0188251 secs] [Times: user=0.05 sys=0.01, real=0.02 secs] 
2019-05-28T18:44:14.984+0000: 2.564: [GC (Allocation Failure) 2019-05-28T18:44:14.984+0000: 2.564: [ParNew: 177505K->12669K(184320K), 0.0897503 secs] 177505K->16962K(1028096K), 0.0898421 secs] [Times: user=0.28 sys=0.02, real=0.09 secs] 
2019-05-28T18:44:15.074+0000: 2.654: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4293K(843776K)] 17954K(1028096K), 0.0028464 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T18:44:15.077+0000: 2.657: [CMS-concurrent-mark-start]
2019-05-28T18:44:15.090+0000: 2.670: [CMS-concurrent-mark: 0.014/0.014 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2019-05-28T18:44:15.090+0000: 2.671: [CMS-concurrent-preclean-start]
2019-05-28T18:44:15.092+0000: 2.672: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-05-28T18:44:15.092+0000: 2.672: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-05-28T18:44:20.172+0000: 7.752: [CMS-concurrent-abortable-preclean: 0.930/5.080 secs] [Times: user=1.45 sys=0.02, real=5.08 secs] 
2019-05-28T18:44:20.172+0000: 7.753: [GC (CMS Final Remark) [YG occupancy: 26397 K (184320 K)]2019-05-28T18:44:20.172+0000: 7.753: [Rescan (parallel) , 0.0020130 secs]2019-05-28T18:44:20.174+0000: 7.755: [weak refs processing, 0.0000217 secs]2019-05-28T18:44:20.174+0000: 7.755: [class unloading, 0.0032350 secs]2019-05-28T18:44:20.178+0000: 7.758: [scrub symbol table, 0.0044977 secs]2019-05-28T18:44:20.182+0000: 7.762: [scrub string table, 0.0004307 secs][1 CMS-remark: 4293K(843776K)] 30691K(1028096K), 0.0105660 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-05-28T18:44:20.183+0000: 7.763: [CMS-concurrent-sweep-start]
2019-05-28T18:44:20.185+0000: 7.766: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-05-28T18:44:20.185+0000: 7.766: [CMS-concurrent-reset-start]
2019-05-28T18:44:20.202+0000: 7.782: [CMS-concurrent-reset: 0.017/0.017 secs] [Times: user=0.00 sys=0.01, real=0.02 secs] 
2019-05-28T19:03:14.608+0000: 1142.188: [GC (Allocation Failure) 2019-05-28T19:03:14.608+0000: 1142.188: [ParNew: 176509K->7397K(184320K), 0.0108119 secs] 180797K->16216K(1028096K), 0.0108977 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 44717K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  22% used [0x00000000c0000000, 0x00000000c2471e18, 0x00000000ca000000)
  from space 20480K,  36% used [0x00000000cb400000, 0x00000000cbb39660, 0x00000000cc800000)
  to   space 20480K,   0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
 concurrent mark-sweep generation total 843776K, used 8818K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 33938K, capacity 34294K, committed 34568K, reserved 1079296K
  class space    used 3861K, capacity 3959K, committed 3960K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906060808 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12254072k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-06T08:08:33.354+0000: 1.725: [GC (Allocation Failure) 2019-06-06T08:08:33.354+0000: 1.725: [ParNew: 163840K->13837K(184320K), 0.0184925 secs] 163840K->13837K(1028096K), 0.0186914 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
2019-06-06T08:08:34.313+0000: 2.683: [GC (Allocation Failure) 2019-06-06T08:08:34.313+0000: 2.683: [ParNew: 177677K->13368K(184320K), 0.0802618 secs] 177677K->17714K(1028096K), 0.0803738 secs] [Times: user=0.22 sys=0.01, real=0.08 secs] 
2019-06-06T08:08:34.394+0000: 2.764: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4346K(843776K)] 18277K(1028096K), 0.0020191 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T08:08:34.396+0000: 2.766: [CMS-concurrent-mark-start]
2019-06-06T08:08:34.410+0000: 2.780: [CMS-concurrent-mark: 0.014/0.014 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2019-06-06T08:08:34.410+0000: 2.780: [CMS-concurrent-preclean-start]
2019-06-06T08:08:34.411+0000: 2.782: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-06T08:08:34.411+0000: 2.782: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-06T08:08:39.449+0000: 7.819: [CMS-concurrent-abortable-preclean: 0.969/5.037 secs] [Times: user=1.83 sys=0.03, real=5.03 secs] 
2019-06-06T08:08:39.449+0000: 7.820: [GC (CMS Final Remark) [YG occupancy: 48656 K (184320 K)]2019-06-06T08:08:39.449+0000: 7.820: [Rescan (parallel) , 0.0047068 secs]2019-06-06T08:08:39.454+0000: 7.824: [weak refs processing, 0.0000261 secs]2019-06-06T08:08:39.454+0000: 7.824: [class unloading, 0.0040219 secs]2019-06-06T08:08:39.458+0000: 7.828: [scrub symbol table, 0.0048993 secs]2019-06-06T08:08:39.463+0000: 7.833: [scrub string table, 0.0004359 secs][1 CMS-remark: 4346K(843776K)] 53002K(1028096K), 0.0146823 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
2019-06-06T08:08:39.464+0000: 7.834: [CMS-concurrent-sweep-start]
2019-06-06T08:08:39.467+0000: 7.837: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T08:08:39.467+0000: 7.837: [CMS-concurrent-reset-start]
2019-06-06T08:08:39.484+0000: 7.854: [CMS-concurrent-reset: 0.017/0.017 secs] [Times: user=0.00 sys=0.02, real=0.02 secs] 
2019-06-06T08:29:03.973+0000: 1232.343: [GC (Allocation Failure) 2019-06-06T08:29:03.973+0000: 1232.343: [ParNew: 177208K->6118K(184320K), 0.0124927 secs] 181548K->14848K(1028096K), 0.0125679 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2019-06-06T09:04:18.305+0000: 3346.675: [GC (Allocation Failure) 2019-06-06T09:04:18.305+0000: 3346.675: [ParNew: 169958K->1886K(184320K), 0.0065652 secs] 178688K->10616K(1028096K), 0.0066572 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 72894K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  43% used [0x00000000c0000000, 0x00000000c4558288, 0x00000000ca000000)
  from space 20480K,   9% used [0x00000000ca000000, 0x00000000ca1d7930, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 8730K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 32758K, capacity 33144K, committed 33416K, reserved 1079296K
  class space    used 3684K, capacity 3788K, committed 3832K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906060928 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12240020k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/gc.log-201906061015 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12234428k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/gc.log-201905281908 <==
2019-06-04T14:14:48.057+0000: 587192.423: [GC (Allocation Failure) 2019-06-04T14:14:48.057+0000: 587192.423: [ParNew: 164430K->581K(184320K), 0.0059328 secs] 175565K->11716K(1028096K), 0.0060199 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-04T15:00:07.232+0000: 589911.598: [GC (Allocation Failure) 2019-06-04T15:00:07.232+0000: 589911.598: [ParNew: 164421K->610K(184320K), 0.0036656 secs] 175556K->11745K(1028096K), 0.0037804 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T15:45:22.231+0000: 592626.597: [GC (Allocation Failure) 2019-06-04T15:45:22.231+0000: 592626.597: [ParNew: 164450K->843K(184320K), 0.0042643 secs] 175585K->11978K(1028096K), 0.0043883 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T16:31:17.211+0000: 595381.577: [GC (Allocation Failure) 2019-06-04T16:31:17.211+0000: 595381.577: [ParNew: 164683K->825K(184320K), 0.0038589 secs] 175818K->11961K(1028096K), 0.0039573 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T17:17:07.382+0000: 598131.748: [GC (Allocation Failure) 2019-06-04T17:17:07.382+0000: 598131.748: [ParNew: 164665K->660K(184320K), 0.0035921 secs] 175801K->11795K(1028096K), 0.0037239 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T18:02:47.209+0000: 600871.575: [GC (Allocation Failure) 2019-06-04T18:02:47.209+0000: 600871.575: [ParNew: 164500K->625K(184320K), 0.0040263 secs] 175635K->11760K(1028096K), 0.0041246 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-04T18:48:37.625+0000: 603621.991: [GC (Allocation Failure) 2019-06-04T18:48:37.625+0000: 603621.991: [ParNew: 164465K->614K(184320K), 0.0041085 secs] 175600K->11749K(1028096K), 0.0042860 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-04T19:33:42.211+0000: 606326.577: [GC (Allocation Failure) 2019-06-04T19:33:42.211+0000: 606326.577: [ParNew: 164454K->584K(184320K), 0.0039405 secs] 175589K->11719K(1028096K), 0.0040320 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T20:19:27.892+0000: 609072.258: [GC (Allocation Failure) 2019-06-04T20:19:27.892+0000: 609072.258: [ParNew: 164424K->547K(184320K), 0.0044761 secs] 175559K->11682K(1028096K), 0.0045977 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T21:05:07.207+0000: 611811.573: [GC (Allocation Failure) 2019-06-04T21:05:07.207+0000: 611811.573: [ParNew: 164387K->599K(184320K), 0.0036573 secs] 175522K->11735K(1028096K), 0.0037448 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T21:49:07.214+0000: 614451.580: [GC (Allocation Failure) 2019-06-04T21:49:07.214+0000: 614451.580: [ParNew: 164439K->532K(184320K), 0.0040360 secs] 175575K->11667K(1028096K), 0.0041259 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T22:36:12.211+0000: 617276.577: [GC (Allocation Failure) 2019-06-04T22:36:12.211+0000: 617276.577: [ParNew: 164372K->611K(184320K), 0.0040272 secs] 175507K->11747K(1028096K), 0.0041649 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-04T23:22:17.209+0000: 620041.575: [GC (Allocation Failure) 2019-06-04T23:22:17.209+0000: 620041.575: [ParNew: 164436K->634K(184320K), 0.0039446 secs] 175572K->11769K(1028096K), 0.0040212 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T00:08:07.975+0000: 622792.341: [GC (Allocation Failure) 2019-06-05T00:08:07.975+0000: 622792.341: [ParNew: 164474K->643K(184320K), 0.0039597 secs] 175609K->11778K(1028096K), 0.0040376 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T00:54:12.210+0000: 625556.576: [GC (Allocation Failure) 2019-06-05T00:54:12.211+0000: 625556.577: [ParNew: 164483K->567K(184320K), 0.0046845 secs] 175618K->11702K(1028096K), 0.0048365 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-05T01:39:48.042+0000: 628292.408: [GC (Allocation Failure) 2019-06-05T01:39:48.042+0000: 628292.408: [ParNew: 164407K->580K(184320K), 0.0045801 secs] 175542K->11716K(1028096K), 0.0046855 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T02:25:48.039+0000: 631052.405: [GC (Allocation Failure) 2019-06-05T02:25:48.039+0000: 631052.405: [ParNew: 164420K->584K(184320K), 0.0053087 secs] 175556K->11720K(1028096K), 0.0053841 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-05T03:12:27.208+0000: 633851.574: [GC (Allocation Failure) 2019-06-05T03:12:27.208+0000: 633851.574: [ParNew: 164424K->594K(184320K), 0.0041572 secs] 175560K->11730K(1028096K), 0.0042699 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-05T03:55:48.038+0000: 636452.404: [GC (Allocation Failure) 2019-06-05T03:55:48.038+0000: 636452.404: [ParNew: 164434K->546K(184320K), 0.0062714 secs] 175570K->11683K(1028096K), 0.0063521 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-05T04:42:59.895+0000: 639284.261: [GC (Allocation Failure) 2019-06-05T04:42:59.895+0000: 639284.261: [ParNew: 164386K->564K(184320K), 0.0040574 secs] 175523K->11700K(1028096K), 0.0041427 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T05:28:48.050+0000: 642032.416: [GC (Allocation Failure) 2019-06-05T05:28:48.050+0000: 642032.416: [ParNew: 164404K->515K(184320K), 0.0040369 secs] 175540K->11651K(1028096K), 0.0041421 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T06:13:43.615+0000: 644727.981: [GC (Allocation Failure) 2019-06-05T06:13:43.615+0000: 644727.981: [ParNew: 164355K->577K(184320K), 0.0038042 secs] 175491K->11713K(1028096K), 0.0039111 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T06:59:35.993+0000: 647480.359: [GC (Allocation Failure) 2019-06-05T06:59:35.994+0000: 647480.360: [ParNew: 164417K->590K(184320K), 0.0047175 secs] 175553K->11726K(1028096K), 0.0048431 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-05T07:44:48.053+0000: 650192.419: [GC (Allocation Failure) 2019-06-05T07:44:48.053+0000: 650192.419: [ParNew: 164430K->653K(184320K), 0.0047845 secs] 175566K->11789K(1028096K), 0.0048913 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-05T08:30:34.708+0000: 652939.074: [GC (Allocation Failure) 2019-06-05T08:30:34.708+0000: 652939.074: [ParNew: 164493K->619K(184320K), 0.0042165 secs] 175629K->11755K(1028096K), 0.0043131 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T09:16:10.425+0000: 655674.791: [GC (Allocation Failure) 2019-06-05T09:16:10.425+0000: 655674.791: [ParNew: 164457K->538K(184320K), 0.0039226 secs] 175594K->11675K(1028096K), 0.0040172 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-05T09:59:48.042+0000: 658292.408: [GC (Allocation Failure) 2019-06-05T09:59:48.042+0000: 658292.408: [ParNew: 164378K->580K(184320K), 0.0100758 secs] 175515K->11716K(1028096K), 0.0101767 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T10:45:32.211+0000: 661036.577: [GC (Allocation Failure) 2019-06-05T10:45:32.211+0000: 661036.577: [ParNew: 164420K->589K(184320K), 0.0043198 secs] 175556K->11726K(1028096K), 0.0044256 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T11:31:12.211+0000: 663776.577: [GC (Allocation Failure) 2019-06-05T11:31:12.211+0000: 663776.577: [ParNew: 164429K->545K(184320K), 0.0044461 secs] 175566K->11682K(1028096K), 0.0045434 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-05T12:15:48.049+0000: 666452.415: [GC (Allocation Failure) 2019-06-05T12:15:48.050+0000: 666452.415: [ParNew: 164385K->581K(184320K), 0.0067979 secs] 175522K->11718K(1028096K), 0.0068820 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T13:01:20.041+0000: 669184.407: [GC (Allocation Failure) 2019-06-05T13:01:20.041+0000: 669184.407: [ParNew: 164421K->569K(184320K), 0.0039660 secs] 175558K->11706K(1028096K), 0.0040517 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T13:47:17.254+0000: 671941.620: [GC (Allocation Failure) 2019-06-05T13:47:17.254+0000: 671941.620: [ParNew: 164409K->609K(184320K), 0.0042510 secs] 175546K->11746K(1028096K), 0.0043727 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
Heap
 par new generation   total 184320K, used 46023K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  27% used [0x00000000c0000000, 0x00000000c2c596a0, 0x00000000ca000000)
  from space 20480K,   2% used [0x00000000cb400000, 0x00000000cb4986a0, 0x00000000cc800000)
  to   space 20480K,   0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
 concurrent mark-sweep generation total 843776K, used 11137K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 37128K, capacity 37566K, committed 37992K, reserved 1083392K
  class space    used 3891K, capacity 4005K, committed 4008K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906060918 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12242580k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-06T09:18:33.287+0000: 1.693: [GC (Allocation Failure) 2019-06-06T09:18:33.287+0000: 1.693: [ParNew: 163840K->13448K(184320K), 0.0164903 secs] 163840K->13448K(1028096K), 0.0166063 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
2019-06-06T09:18:35.304+0000: 3.709: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 134939K(1028096K), 0.0106761 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2019-06-06T09:18:35.315+0000: 3.720: [CMS-concurrent-mark-start]
2019-06-06T09:18:35.322+0000: 3.728: [CMS-concurrent-mark: 0.008/0.008 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-06T09:18:35.323+0000: 3.728: [CMS-concurrent-preclean-start]
2019-06-06T09:18:35.324+0000: 3.729: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-06T09:18:35.324+0000: 3.729: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-06T09:18:40.413+0000: 8.818: [CMS-concurrent-abortable-preclean: 1.268/5.089 secs] [Times: user=1.28 sys=0.00, real=5.09 secs] 
2019-06-06T09:18:40.413+0000: 8.819: [GC (CMS Final Remark) [YG occupancy: 138216 K (184320 K)]2019-06-06T09:18:40.413+0000: 8.819: [Rescan (parallel) , 0.0096532 secs]2019-06-06T09:18:40.423+0000: 8.828: [weak refs processing, 0.0000214 secs]2019-06-06T09:18:40.423+0000: 8.828: [class unloading, 0.0032172 secs]2019-06-06T09:18:40.426+0000: 8.832: [scrub symbol table, 0.0039939 secs]2019-06-06T09:18:40.430+0000: 8.836: [scrub string table, 0.0003468 secs][1 CMS-remark: 0K(843776K)] 138216K(1028096K), 0.0178463 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 
2019-06-06T09:18:40.431+0000: 8.837: [CMS-concurrent-sweep-start]
2019-06-06T09:18:40.431+0000: 8.837: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-06T09:18:40.431+0000: 8.837: [CMS-concurrent-reset-start]
2019-06-06T09:18:40.448+0000: 8.854: [CMS-concurrent-reset: 0.017/0.017 secs] [Times: user=0.00 sys=0.02, real=0.01 secs] 
2019-06-06T09:20:53.929+0000: 142.334: [GC (Allocation Failure) 2019-06-06T09:20:53.929+0000: 142.334: [ParNew: 177288K->13333K(184320K), 0.0388667 secs] 177288K->17936K(1028096K), 0.0389821 secs] [Times: user=0.09 sys=0.03, real=0.04 secs] 
Heap
 par new generation   total 184320K, used 46620K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  20% used [0x00000000c0000000, 0x00000000c2081ba0, 0x00000000ca000000)
  from space 20480K,  65% used [0x00000000ca000000, 0x00000000cad05590, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4603K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 28734K, capacity 29036K, committed 29348K, reserved 1075200K
  class space    used 3366K, capacity 3471K, committed 3552K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906061011 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12235200k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev-hdfs-restore-clstr-w1.out.1 <==
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 55973
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201906051358 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12069472k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T13:58:40.608+0000: 1.791: [GC (Allocation Failure) 2019-06-05T13:58:40.608+0000: 1.791: [ParNew: 163840K->13768K(184320K), 0.0184207 secs] 163840K->13768K(1028096K), 0.0185598 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
2019-06-05T13:58:41.618+0000: 2.800: [GC (Allocation Failure) 2019-06-05T13:58:41.618+0000: 2.801: [ParNew: 177608K->12366K(184320K), 0.0633344 secs] 177608K->16699K(1028096K), 0.0634443 secs] [Times: user=0.15 sys=0.01, real=0.07 secs] 
2019-06-05T13:58:41.685+0000: 2.867: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4333K(843776K)] 19598K(1028096K), 0.0036717 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T13:58:41.688+0000: 2.871: [CMS-concurrent-mark-start]
2019-06-05T13:58:41.706+0000: 2.889: [CMS-concurrent-mark: 0.018/0.018 secs] [Times: user=0.06 sys=0.02, real=0.02 secs] 
2019-06-05T13:58:41.706+0000: 2.889: [CMS-concurrent-preclean-start]
2019-06-05T13:58:41.708+0000: 2.891: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T13:58:41.708+0000: 2.891: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T13:58:46.711+0000: 7.894: [CMS-concurrent-abortable-preclean: 1.058/5.003 secs] [Times: user=2.02 sys=0.04, real=5.01 secs] 
2019-06-05T13:58:46.711+0000: 7.894: [GC (CMS Final Remark) [YG occupancy: 46068 K (184320 K)]2019-06-05T13:58:46.711+0000: 7.894: [Rescan (parallel) , 0.0041329 secs]2019-06-05T13:58:46.716+0000: 7.898: [weak refs processing, 0.0000277 secs]2019-06-05T13:58:46.716+0000: 7.898: [class unloading, 0.0035582 secs]2019-06-05T13:58:46.719+0000: 7.902: [scrub symbol table, 0.0050431 secs]2019-06-05T13:58:46.724+0000: 7.907: [scrub string table, 0.0005514 secs][1 CMS-remark: 4333K(843776K)] 50402K(1028096K), 0.0139301 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2019-06-05T13:58:46.726+0000: 7.908: [CMS-concurrent-sweep-start]
2019-06-05T13:58:46.737+0000: 7.920: [CMS-concurrent-sweep: 0.011/0.011 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T13:58:46.737+0000: 7.920: [CMS-concurrent-reset-start]
2019-06-05T13:58:46.748+0000: 7.931: [CMS-concurrent-reset: 0.011/0.011 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 115834K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  63% used [0x00000000c0000000, 0x00000000c650b2c0, 0x00000000ca000000)
  from space 20480K,  60% used [0x00000000ca000000, 0x00000000cac13848, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4327K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 32904K, capacity 33252K, committed 33896K, reserved 1079296K
  class space    used 3799K, capacity 3892K, committed 4012K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev-hdfs-restore-clstr-w1.out.5 <==
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 55973
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201906061017 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12234008k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-06T10:17:15.591+0000: 1.747: [GC (Allocation Failure) 2019-06-06T10:17:15.591+0000: 1.747: [ParNew: 163840K->13449K(184320K), 0.0174819 secs] 163840K->13449K(1028096K), 0.0176175 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2019-06-06T10:17:16.665+0000: 2.821: [GC (Allocation Failure) 2019-06-06T10:17:16.665+0000: 2.821: [ParNew: 177289K->14373K(184320K), 0.0651197 secs] 177289K->18570K(1028096K), 0.0652282 secs] [Times: user=0.15 sys=0.02, real=0.06 secs] 
2019-06-06T10:17:16.732+0000: 2.887: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4196K(843776K)] 19089K(1028096K), 0.0036087 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T10:17:16.735+0000: 2.891: [CMS-concurrent-mark-start]
2019-06-06T10:17:16.754+0000: 2.910: [CMS-concurrent-mark: 0.019/0.019 secs] [Times: user=0.07 sys=0.01, real=0.02 secs] 
2019-06-06T10:17:16.754+0000: 2.910: [CMS-concurrent-preclean-start]
2019-06-06T10:17:16.756+0000: 2.912: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-06T10:17:16.756+0000: 2.912: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-06T10:17:21.876+0000: 8.032: [CMS-concurrent-abortable-preclean: 1.062/5.121 secs] [Times: user=1.94 sys=0.03, real=5.12 secs] 
2019-06-06T10:17:21.877+0000: 8.033: [GC (CMS Final Remark) [YG occupancy: 40998 K (184320 K)]2019-06-06T10:17:21.877+0000: 8.033: [Rescan (parallel) , 0.0038363 secs]2019-06-06T10:17:21.881+0000: 8.037: [weak refs processing, 0.0000235 secs]2019-06-06T10:17:21.881+0000: 8.037: [class unloading, 0.0038787 secs]2019-06-06T10:17:21.885+0000: 8.041: [scrub symbol table, 0.0049732 secs]2019-06-06T10:17:21.890+0000: 8.045: [scrub string table, 0.0005293 secs][1 CMS-remark: 4196K(843776K)] 45194K(1028096K), 0.0140083 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
2019-06-06T10:17:21.891+0000: 8.047: [CMS-concurrent-sweep-start]
2019-06-06T10:17:21.894+0000: 8.049: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-06T10:17:21.894+0000: 8.049: [CMS-concurrent-reset-start]
2019-06-06T10:17:21.910+0000: 8.066: [CMS-concurrent-reset: 0.017/0.017 secs] [Times: user=0.00 sys=0.02, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 170503K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  95% used [0x00000000c0000000, 0x00000000c98787c8, 0x00000000ca000000)
  from space 20480K,  70% used [0x00000000ca000000, 0x00000000cae09568, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4191K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 33193K, capacity 33546K, committed 33980K, reserved 1079296K
  class space    used 3767K, capacity 3861K, committed 3884K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev-hdfs-restore-clstr-w1.out.4 <==
Error opening zip file or JAR manifest missing : /home/sshhdfsuser/jmx_prometheus_javaagent-0.11.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 55973
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201906051403 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12280652k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T14:03:17.038+0000: 1.733: [GC (Allocation Failure) 2019-06-05T14:03:17.038+0000: 1.733: [ParNew: 163840K->13452K(184320K), 0.0171423 secs] 163840K->13452K(1028096K), 0.0173024 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
2019-06-05T14:03:18.089+0000: 2.784: [GC (Allocation Failure) 2019-06-05T14:03:18.089+0000: 2.784: [ParNew: 177292K->13390K(184320K), 0.0715493 secs] 177292K->17584K(1028096K), 0.0716614 secs] [Times: user=0.18 sys=0.01, real=0.07 secs] 
2019-06-05T14:03:18.161+0000: 2.856: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4194K(843776K)] 18103K(1028096K), 0.0027776 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T14:03:18.164+0000: 2.858: [CMS-concurrent-mark-start]
2019-06-05T14:03:18.177+0000: 2.872: [CMS-concurrent-mark: 0.014/0.014 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
2019-06-05T14:03:18.178+0000: 2.872: [CMS-concurrent-preclean-start]
2019-06-05T14:03:18.179+0000: 2.874: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T14:03:18.179+0000: 2.874: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T14:03:23.190+0000: 7.885: [CMS-concurrent-abortable-preclean: 1.049/5.011 secs] [Times: user=1.84 sys=0.03, real=5.02 secs] 
2019-06-05T14:03:23.191+0000: 7.886: [GC (CMS Final Remark) [YG occupancy: 43753 K (184320 K)]2019-06-05T14:03:23.191+0000: 7.886: [Rescan (parallel) , 0.0038170 secs]2019-06-05T14:03:23.195+0000: 7.890: [weak refs processing, 0.0000294 secs]2019-06-05T14:03:23.195+0000: 7.890: [class unloading, 0.0036481 secs]2019-06-05T14:03:23.198+0000: 7.893: [scrub symbol table, 0.0046862 secs]2019-06-05T14:03:23.203+0000: 7.898: [scrub string table, 0.0005318 secs][1 CMS-remark: 4194K(843776K)] 47947K(1028096K), 0.0133000 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-05T14:03:23.204+0000: 7.899: [CMS-concurrent-sweep-start]
2019-06-05T14:03:23.207+0000: 7.902: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T14:03:23.207+0000: 7.902: [CMS-concurrent-reset-start]
2019-06-05T14:03:23.222+0000: 7.917: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.01 sys=0.02, real=0.02 secs] 
Heap
 par new generation   total 184320K, used 137068K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  75% used [0x00000000c0000000, 0x00000000c78c7b58, 0x00000000ca000000)
  from space 20480K,  65% used [0x00000000ca000000, 0x00000000cad13860, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4188K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 32156K, capacity 32522K, committed 33000K, reserved 1079296K
  class space    used 3681K, capacity 3797K, committed 3884K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906051603 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12267236k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T16:03:39.984+0000: 1.740: [GC (Allocation Failure) 2019-06-05T16:03:39.984+0000: 1.740: [ParNew: 163840K->13449K(184320K), 0.0172077 secs] 163840K->13449K(1028096K), 0.0173414 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
2019-06-05T16:03:41.050+0000: 2.807: [GC (Allocation Failure) 2019-06-05T16:03:41.050+0000: 2.807: [ParNew: 177289K->12372K(184320K), 0.0624747 secs] 177289K->16573K(1028096K), 0.0625895 secs] [Times: user=0.14 sys=0.01, real=0.07 secs] 
2019-06-05T16:03:41.116+0000: 2.873: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4201K(843776K)] 18131K(1028096K), 0.0032280 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T16:03:41.123+0000: 2.879: [CMS-concurrent-mark-start]
2019-06-05T16:03:41.138+0000: 2.895: [CMS-concurrent-mark: 0.015/0.015 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2019-06-05T16:03:41.138+0000: 2.895: [CMS-concurrent-preclean-start]
2019-06-05T16:03:41.140+0000: 2.897: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T16:03:41.140+0000: 2.897: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T16:03:46.265+0000: 8.022: [CMS-concurrent-abortable-preclean: 1.077/5.125 secs] [Times: user=1.87 sys=0.02, real=5.13 secs] 
2019-06-05T16:03:46.266+0000: 8.022: [GC (CMS Final Remark) [YG occupancy: 39540 K (184320 K)]2019-06-05T16:03:46.266+0000: 8.022: [Rescan (parallel) , 0.0037864 secs]2019-06-05T16:03:46.270+0000: 8.026: [weak refs processing, 0.0000237 secs]2019-06-05T16:03:46.270+0000: 8.026: [class unloading, 0.0124953 secs]2019-06-05T16:03:46.282+0000: 8.039: [scrub symbol table, 0.0047589 secs]2019-06-05T16:03:46.287+0000: 8.044: [scrub string table, 0.0005898 secs][1 CMS-remark: 4201K(843776K)] 43742K(1028096K), 0.0222743 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
2019-06-05T16:03:46.288+0000: 8.045: [CMS-concurrent-sweep-start]
2019-06-05T16:03:46.291+0000: 8.047: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
2019-06-05T16:03:46.291+0000: 8.047: [CMS-concurrent-reset-start]
2019-06-05T16:03:46.307+0000: 8.064: [CMS-concurrent-reset: 0.017/0.017 secs] [Times: user=0.00 sys=0.02, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 81967K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  42% used [0x00000000c0000000, 0x00000000c43f6ce0, 0x00000000ca000000)
  from space 20480K,  60% used [0x00000000ca000000, 0x00000000cac15168, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4195K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 31229K, capacity 31576K, committed 31976K, reserved 1077248K
  class space    used 3645K, capacity 3762K, committed 3884K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906060938 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12238044k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/gc.log-201906061037 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12222536k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/gc.log-201906051418 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12279756k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T14:18:47.141+0000: 1.768: [GC (Allocation Failure) 2019-06-05T14:18:47.141+0000: 1.768: [ParNew: 163840K->13448K(184320K), 0.0203847 secs] 163840K->13448K(1028096K), 0.0205288 secs] [Times: user=0.04 sys=0.01, real=0.02 secs] 
2019-06-05T14:18:48.216+0000: 2.844: [GC (Allocation Failure) 2019-06-05T14:18:48.217+0000: 2.844: [ParNew: 177288K->15140K(184320K), 0.0823561 secs] 177288K->19338K(1028096K), 0.0825264 secs] [Times: user=0.27 sys=0.02, real=0.08 secs] 
2019-06-05T14:18:48.304+0000: 2.932: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4197K(843776K)] 19751K(1028096K), 0.0033690 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T14:18:48.307+0000: 2.935: [CMS-concurrent-mark-start]
2019-06-05T14:18:48.321+0000: 2.949: [CMS-concurrent-mark: 0.014/0.014 secs] [Times: user=0.05 sys=0.01, real=0.02 secs] 
2019-06-05T14:18:48.321+0000: 2.949: [CMS-concurrent-preclean-start]
2019-06-05T14:18:48.323+0000: 2.951: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T14:18:48.324+0000: 2.952: [GC (CMS Final Remark) [YG occupancy: 16610 K (184320 K)]2019-06-05T14:18:48.324+0000: 2.952: [Rescan (parallel) , 0.0058850 secs]2019-06-05T14:18:48.330+0000: 2.957: [weak refs processing, 0.0000231 secs]2019-06-05T14:18:48.330+0000: 2.957: [class unloading, 0.0037305 secs]2019-06-05T14:18:48.333+0000: 2.961: [scrub symbol table, 0.0045633 secs]2019-06-05T14:18:48.338+0000: 2.966: [scrub string table, 0.0004728 secs][1 CMS-remark: 4197K(843776K)] 20808K(1028096K), 0.0151944 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2019-06-05T14:18:48.339+0000: 2.967: [CMS-concurrent-sweep-start]
2019-06-05T14:18:48.342+0000: 2.970: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T14:18:48.342+0000: 2.970: [CMS-concurrent-reset-start]
2019-06-05T14:18:48.372+0000: 3.000: [CMS-concurrent-reset: 0.030/0.030 secs] [Times: user=0.09 sys=0.02, real=0.03 secs] 
Heap
 par new generation   total 184320K, used 103061K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  53% used [0x00000000c0000000, 0x00000000c55dc180, 0x00000000ca000000)
  from space 20480K,  73% used [0x00000000ca000000, 0x00000000caec92a0, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4191K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 32328K, capacity 32708K, committed 33032K, reserved 1079296K
  class space    used 3752K, capacity 3860K, committed 3960K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906051556 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12273688k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T15:56:48.642+0000: 1.736: [GC (Allocation Failure) 2019-06-05T15:56:48.642+0000: 1.736: [ParNew: 163840K->13459K(184320K), 0.0167936 secs] 163840K->13459K(1028096K), 0.0169458 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2019-06-05T15:56:49.704+0000: 2.798: [GC (Allocation Failure) 2019-06-05T15:56:49.704+0000: 2.798: [ParNew: 177299K->12387K(184320K), 0.0642902 secs] 177299K->16583K(1028096K), 0.0644002 secs] [Times: user=0.15 sys=0.01, real=0.06 secs] 
2019-06-05T15:56:49.771+0000: 2.865: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4195K(843776K)] 18141K(1028096K), 0.0034440 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T15:56:49.775+0000: 2.869: [CMS-concurrent-mark-start]
2019-06-05T15:56:49.790+0000: 2.884: [CMS-concurrent-mark: 0.015/0.015 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
2019-06-05T15:56:49.790+0000: 2.884: [CMS-concurrent-preclean-start]
2019-06-05T15:56:49.792+0000: 2.886: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
2019-06-05T15:56:49.792+0000: 2.886: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T15:56:54.826+0000: 7.920: [CMS-concurrent-abortable-preclean: 1.046/5.034 secs] [Times: user=1.80 sys=0.03, real=5.03 secs] 
2019-06-05T15:56:54.827+0000: 7.921: [GC (CMS Final Remark) [YG occupancy: 41086 K (184320 K)]2019-06-05T15:56:54.827+0000: 7.921: [Rescan (parallel) , 0.0040711 secs]2019-06-05T15:56:54.831+0000: 7.925: [weak refs processing, 0.0000252 secs]2019-06-05T15:56:54.831+0000: 7.925: [class unloading, 0.0039409 secs]2019-06-05T15:56:54.835+0000: 7.929: [scrub symbol table, 0.0048095 secs]2019-06-05T15:56:54.839+0000: 7.934: [scrub string table, 0.0004975 secs][1 CMS-remark: 4195K(843776K)] 45282K(1028096K), 0.0140144 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
2019-06-05T15:56:54.841+0000: 7.935: [CMS-concurrent-sweep-start]
2019-06-05T15:56:54.843+0000: 7.937: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T15:56:54.843+0000: 7.937: [CMS-concurrent-reset-start]
2019-06-05T15:56:54.858+0000: 7.952: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 113831K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  61% used [0x00000000c0000000, 0x00000000c6311080, 0x00000000ca000000)
  from space 20480K,  60% used [0x00000000ca000000, 0x00000000cac18f00, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4190K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 32644K, capacity 32994K, committed 33212K, reserved 1079296K
  class space    used 3765K, capacity 3865K, committed 3884K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906051609 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12256264k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T16:09:31.019+0000: 1.761: [GC (Allocation Failure) 2019-06-05T16:09:31.019+0000: 1.761: [ParNew: 163840K->13455K(184320K), 0.0181059 secs] 163840K->13455K(1028096K), 0.0182205 secs] [Times: user=0.04 sys=0.01, real=0.02 secs] 
2019-06-05T16:09:33.043+0000: 3.785: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 134938K(1028096K), 0.0111379 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2019-06-05T16:09:33.054+0000: 3.796: [CMS-concurrent-mark-start]
2019-06-05T16:09:33.062+0000: 3.804: [CMS-concurrent-mark: 0.008/0.008 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T16:09:33.062+0000: 3.804: [CMS-concurrent-preclean-start]
2019-06-05T16:09:33.064+0000: 3.806: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T16:09:33.064+0000: 3.806: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T16:09:38.082+0000: 8.823: [CMS-concurrent-abortable-preclean: 1.278/5.017 secs] [Times: user=1.29 sys=0.00, real=5.02 secs] 
2019-06-05T16:09:38.082+0000: 8.824: [GC (CMS Final Remark) [YG occupancy: 138215 K (184320 K)]2019-06-05T16:09:38.082+0000: 8.824: [Rescan (parallel) , 0.0113016 secs]2019-06-05T16:09:38.093+0000: 8.835: [weak refs processing, 0.0000198 secs]2019-06-05T16:09:38.093+0000: 8.835: [class unloading, 0.0032654 secs]2019-06-05T16:09:38.096+0000: 8.838: [scrub symbol table, 0.0042903 secs]2019-06-05T16:09:38.101+0000: 8.843: [scrub string table, 0.0003524 secs][1 CMS-remark: 0K(843776K)] 138215K(1028096K), 0.0198484 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 
2019-06-05T16:09:38.102+0000: 8.844: [CMS-concurrent-sweep-start]
2019-06-05T16:09:38.102+0000: 8.844: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T16:09:38.102+0000: 8.844: [CMS-concurrent-reset-start]
2019-06-05T16:09:38.115+0000: 8.857: [CMS-concurrent-reset: 0.013/0.013 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] 
2019-06-05T16:11:51.006+0000: 141.748: [GC (Allocation Failure) 2019-06-05T16:11:51.006+0000: 141.748: [ParNew: 177295K->13349K(184320K), 0.0427774 secs] 177295K->17915K(1028096K), 0.0428907 secs] [Times: user=0.11 sys=0.02, real=0.04 secs] 
Heap
 par new generation   total 184320K, used 103236K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  54% used [0x00000000c0000000, 0x00000000c57c7b28, 0x00000000ca000000)
  from space 20480K,  65% used [0x00000000ca000000, 0x00000000cad09768, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4565K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 31384K, capacity 31762K, committed 32140K, reserved 1077248K
  class space    used 3637K, capacity 3760K, committed 3784K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906060927 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12241096k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/gc.log-201906061006 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12235576k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-06T10:06:57.080+0000: 1.749: [GC (Allocation Failure) 2019-06-06T10:06:57.081+0000: 1.750: [ParNew: 163840K->13854K(184320K), 0.0315588 secs] 163840K->13854K(1028096K), 0.0317086 secs] [Times: user=0.10 sys=0.02, real=0.03 secs] 
2019-06-06T10:06:59.112+0000: 3.781: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 143170K(1028096K), 0.0099077 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2019-06-06T10:06:59.122+0000: 3.791: [CMS-concurrent-mark-start]
2019-06-06T10:06:59.130+0000: 3.799: [CMS-concurrent-mark: 0.008/0.008 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T10:06:59.130+0000: 3.799: [CMS-concurrent-preclean-start]
2019-06-06T10:06:59.131+0000: 3.800: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
2019-06-06T10:06:59.131+0000: 3.800: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-06T10:07:04.170+0000: 8.839: [CMS-concurrent-abortable-preclean: 1.305/5.039 secs] [Times: user=1.32 sys=0.00, real=5.03 secs] 
2019-06-06T10:07:04.170+0000: 8.839: [GC (CMS Final Remark) [YG occupancy: 146447 K (184320 K)]2019-06-06T10:07:04.170+0000: 8.839: [Rescan (parallel) , 0.0099437 secs]2019-06-06T10:07:04.180+0000: 8.849: [weak refs processing, 0.0000210 secs]2019-06-06T10:07:04.180+0000: 8.849: [class unloading, 0.0035093 secs]2019-06-06T10:07:04.184+0000: 8.853: [scrub symbol table, 0.0040732 secs]2019-06-06T10:07:04.188+0000: 8.857: [scrub string table, 0.0003550 secs][1 CMS-remark: 0K(843776K)] 146447K(1028096K), 0.0185656 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 
2019-06-06T10:07:04.189+0000: 8.858: [CMS-concurrent-sweep-start]
2019-06-06T10:07:04.189+0000: 8.858: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-06T10:07:04.189+0000: 8.858: [CMS-concurrent-reset-start]
2019-06-06T10:07:04.207+0000: 8.876: [CMS-concurrent-reset: 0.018/0.018 secs] [Times: user=0.00 sys=0.02, real=0.02 secs] 
2019-06-06T10:07:07.177+0000: 11.846: [GC (Allocation Failure) 2019-06-06T10:07:07.177+0000: 11.846: [ParNew: 177694K->11122K(184320K), 0.0649732 secs] 177694K->15896K(1028096K), 0.0650911 secs] [Times: user=0.16 sys=0.01, real=0.07 secs] 
Heap
 par new generation   total 184320K, used 97456K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  52% used [0x00000000c0000000, 0x00000000c544f9b0, 0x00000000ca000000)
  from space 20480K,  54% used [0x00000000ca000000, 0x00000000caadc9b0, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4773K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 31579K, capacity 32016K, committed 32444K, reserved 1077248K
  class space    used 3671K, capacity 3798K, committed 3912K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906051421 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12279500k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T14:21:41.395+0000: 1.748: [GC (Allocation Failure) 2019-06-05T14:21:41.395+0000: 1.748: [ParNew: 163840K->13456K(184320K), 0.0166494 secs] 163840K->13456K(1028096K), 0.0167751 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
2019-06-05T14:21:42.425+0000: 2.778: [GC (Allocation Failure) 2019-06-05T14:21:42.425+0000: 2.778: [ParNew: 177296K->13382K(184320K), 0.0571978 secs] 177296K->17578K(1028096K), 0.0573147 secs] [Times: user=0.13 sys=0.02, real=0.06 secs] 
2019-06-05T14:21:42.483+0000: 2.836: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4195K(843776K)] 18097K(1028096K), 0.0028181 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T14:21:42.486+0000: 2.839: [CMS-concurrent-mark-start]
2019-06-05T14:21:42.500+0000: 2.853: [CMS-concurrent-mark: 0.014/0.014 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
2019-06-05T14:21:42.500+0000: 2.853: [CMS-concurrent-preclean-start]
2019-06-05T14:21:42.502+0000: 2.855: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T14:21:42.502+0000: 2.855: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T14:21:47.542+0000: 7.895: [CMS-concurrent-abortable-preclean: 1.074/5.040 secs] [Times: user=1.85 sys=0.02, real=5.04 secs] 
2019-06-05T14:21:47.542+0000: 7.895: [GC (CMS Final Remark) [YG occupancy: 42078 K (184320 K)]2019-06-05T14:21:47.542+0000: 7.895: [Rescan (parallel) , 0.0040662 secs]2019-06-05T14:21:47.546+0000: 7.899: [weak refs processing, 0.0000299 secs]2019-06-05T14:21:47.546+0000: 7.899: [class unloading, 0.0036834 secs]2019-06-05T14:21:47.550+0000: 7.903: [scrub symbol table, 0.0050422 secs]2019-06-05T14:21:47.555+0000: 7.908: [scrub string table, 0.0005772 secs][1 CMS-remark: 4195K(843776K)] 46274K(1028096K), 0.0139443 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2019-06-05T14:21:47.557+0000: 7.910: [CMS-concurrent-sweep-start]
2019-06-05T14:21:47.559+0000: 7.912: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T14:21:47.560+0000: 7.912: [CMS-concurrent-reset-start]
2019-06-05T14:21:47.586+0000: 7.938: [CMS-concurrent-reset: 0.026/0.026 secs] [Times: user=0.01 sys=0.02, real=0.03 secs] 
2019-06-05T14:39:02.047+0000: 1042.400: [GC (Allocation Failure) 2019-06-05T14:39:02.047+0000: 1042.400: [ParNew: 177222K->7221K(184320K), 0.0124683 secs] 181412K->15914K(1028096K), 0.0125835 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2019-06-05T15:17:26.383+0000: 3346.736: [GC (Allocation Failure) 2019-06-05T15:17:26.383+0000: 3346.736: [ParNew: 171061K->2070K(184320K), 0.0050704 secs] 179754K->10763K(1028096K), 0.0052097 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
Heap
 par new generation   total 184320K, used 80140K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  47% used [0x00000000c0000000, 0x00000000c4c3d6f0, 0x00000000ca000000)
  from space 20480K,  10% used [0x00000000ca000000, 0x00000000ca2059f0, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 8693K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 34136K, capacity 34484K, committed 34792K, reserved 1081344K
  class space    used 3766K, capacity 3878K, committed 3880K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906060925 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12242536k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/gc.log-201906060942 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12236944k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-06T09:42:11.158+0000: 1.721: [GC (Allocation Failure) 2019-06-06T09:42:11.158+0000: 1.721: [ParNew: 163840K->13445K(184320K), 0.0163201 secs] 163840K->13445K(1028096K), 0.0164447 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
2019-06-06T09:42:12.256+0000: 2.819: [GC (Allocation Failure) 2019-06-06T09:42:12.256+0000: 2.819: [ParNew: 177285K->12400K(184320K), 0.0591132 secs] 177285K->16597K(1028096K), 0.0592245 secs] [Times: user=0.16 sys=0.02, real=0.06 secs] 
2019-06-06T09:42:12.317+0000: 2.880: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4197K(843776K)] 18668K(1028096K), 0.0031019 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T09:42:12.326+0000: 2.889: [CMS-concurrent-mark-start]
2019-06-06T09:42:12.343+0000: 2.906: [CMS-concurrent-mark: 0.017/0.017 secs] [Times: user=0.06 sys=0.01, real=0.02 secs] 
2019-06-06T09:42:12.344+0000: 2.906: [CMS-concurrent-preclean-start]
2019-06-06T09:42:12.345+0000: 2.908: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-06T09:42:12.345+0000: 2.908: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-06T09:42:17.456+0000: 8.019: [CMS-concurrent-abortable-preclean: 1.046/5.111 secs] [Times: user=1.74 sys=0.02, real=5.11 secs] 
2019-06-06T09:42:17.457+0000: 8.020: [GC (CMS Final Remark) [YG occupancy: 39228 K (184320 K)]2019-06-06T09:42:17.457+0000: 8.020: [Rescan (parallel) , 0.0045660 secs]2019-06-06T09:42:17.461+0000: 8.024: [weak refs processing, 0.0000274 secs]2019-06-06T09:42:17.461+0000: 8.024: [class unloading, 0.0036341 secs]2019-06-06T09:42:17.465+0000: 8.028: [scrub symbol table, 0.0044451 secs]2019-06-06T09:42:17.469+0000: 8.032: [scrub string table, 0.0004436 secs][1 CMS-remark: 4197K(843776K)] 43425K(1028096K), 0.0135787 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-06T09:42:17.470+0000: 8.033: [CMS-concurrent-sweep-start]
2019-06-06T09:42:17.473+0000: 8.036: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T09:42:17.473+0000: 8.036: [CMS-concurrent-reset-start]
2019-06-06T09:42:17.492+0000: 8.055: [CMS-concurrent-reset: 0.019/0.019 secs] [Times: user=0.01 sys=0.02, real=0.02 secs] 
2019-06-06T10:01:31.819+0000: 1162.382: [GC (Allocation Failure) 2019-06-06T10:01:31.819+0000: 1162.382: [ParNew: 176240K->7186K(184320K), 0.0118311 secs] 180432K->15965K(1028096K), 0.0119638 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
Heap
 par new generation   total 184320K, used 41711K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  21% used [0x00000000c0000000, 0x00000000c21b7410, 0x00000000ca000000)
  from space 20480K,  35% used [0x00000000cb400000, 0x00000000cbb04a10, 0x00000000cc800000)
  to   space 20480K,   0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
 concurrent mark-sweep generation total 843776K, used 8779K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 33103K, capacity 33500K, committed 33848K, reserved 1079296K
  class space    used 3766K, capacity 3864K, committed 3984K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906051535 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12278336k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T15:35:07.811+0000: 1.703: [GC (Allocation Failure) 2019-06-05T15:35:07.811+0000: 1.703: [ParNew: 163840K->13442K(184320K), 0.0173424 secs] 163840K->13442K(1028096K), 0.0174581 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2019-06-05T15:35:08.865+0000: 2.757: [GC (Allocation Failure) 2019-06-05T15:35:08.865+0000: 2.757: [ParNew: 177282K->12411K(184320K), 0.0746993 secs] 177282K->16605K(1028096K), 0.0748099 secs] [Times: user=0.18 sys=0.01, real=0.07 secs] 
2019-06-05T15:35:08.940+0000: 2.832: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4194K(843776K)] 17127K(1028096K), 0.0025584 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 
2019-06-05T15:35:08.943+0000: 2.834: [CMS-concurrent-mark-start]
2019-06-05T15:35:08.965+0000: 2.857: [CMS-concurrent-mark: 0.022/0.022 secs] [Times: user=0.06 sys=0.02, real=0.02 secs] 
2019-06-05T15:35:08.965+0000: 2.857: [CMS-concurrent-preclean-start]
2019-06-05T15:35:08.967+0000: 2.858: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T15:35:08.967+0000: 2.858: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T15:35:14.000+0000: 7.892: [CMS-concurrent-abortable-preclean: 1.051/5.033 secs] [Times: user=1.91 sys=0.02, real=5.03 secs] 
2019-06-05T15:35:14.001+0000: 7.892: [GC (CMS Final Remark) [YG occupancy: 41175 K (184320 K)]2019-06-05T15:35:14.001+0000: 7.892: [Rescan (parallel) , 0.0039866 secs]2019-06-05T15:35:14.005+0000: 7.896: [weak refs processing, 0.0000284 secs]2019-06-05T15:35:14.005+0000: 7.896: [class unloading, 0.0036145 secs]2019-06-05T15:35:14.008+0000: 7.900: [scrub symbol table, 0.0047300 secs]2019-06-05T15:35:14.013+0000: 7.905: [scrub string table, 0.0005356 secs][1 CMS-remark: 4194K(843776K)] 45370K(1028096K), 0.0135511 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2019-06-05T15:35:14.015+0000: 7.906: [CMS-concurrent-sweep-start]
2019-06-05T15:35:14.017+0000: 7.909: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T15:35:14.018+0000: 7.909: [CMS-concurrent-reset-start]
2019-06-05T15:35:14.041+0000: 7.932: [CMS-concurrent-reset: 0.023/0.023 secs] [Times: user=0.01 sys=0.02, real=0.03 secs] 
Heap
 par new generation   total 184320K, used 119065K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  65% used [0x00000000c0000000, 0x00000000c68276a8, 0x00000000ca000000)
  from space 20480K,  60% used [0x00000000ca000000, 0x00000000cac1edf8, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4188K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 32676K, capacity 33034K, committed 33568K, reserved 1079296K
  class space    used 3764K, capacity 3861K, committed 3960K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906061013 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12234652k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev-hdfs-restore-clstr-w1.out.3 <==
Error opening zip file or JAR manifest missing : /home/sshhdfsuser/jmx_prometheus_javaagent-0.11.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 55973
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201906051543 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12275064k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T15:43:45.836+0000: 1.763: [GC (Allocation Failure) 2019-06-05T15:43:45.836+0000: 1.763: [ParNew: 163840K->13440K(184320K), 0.0163099 secs] 163840K->13440K(1028096K), 0.0164814 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
2019-06-05T15:43:46.899+0000: 2.826: [GC (Allocation Failure) 2019-06-05T15:43:46.899+0000: 2.826: [ParNew: 177280K->13370K(184320K), 0.0695785 secs] 177280K->17565K(1028096K), 0.0696872 secs] [Times: user=0.17 sys=0.01, real=0.07 secs] 
2019-06-05T15:43:46.972+0000: 2.899: [GC (CMS Initial Mark) [1 CMS-initial-mark: 4195K(843776K)] 19123K(1028096K), 0.0021818 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T15:43:46.974+0000: 2.901: [CMS-concurrent-mark-start]
2019-06-05T15:43:46.989+0000: 2.916: [CMS-concurrent-mark: 0.014/0.014 secs] [Times: user=0.04 sys=0.02, real=0.01 secs] 
2019-06-05T15:43:46.989+0000: 2.916: [CMS-concurrent-preclean-start]
2019-06-05T15:43:46.990+0000: 2.917: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-05T15:43:46.990+0000: 2.917: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T15:43:52.105+0000: 8.032: [CMS-concurrent-abortable-preclean: 1.117/5.114 secs] [Times: user=2.31 sys=0.03, real=5.12 secs] 
2019-06-05T15:43:52.105+0000: 8.032: [GC (CMS Final Remark) [YG occupancy: 52566 K (184320 K)]2019-06-05T15:43:52.105+0000: 8.032: [Rescan (parallel) , 0.0042082 secs]2019-06-05T15:43:52.109+0000: 8.036: [weak refs processing, 0.0000248 secs]2019-06-05T15:43:52.109+0000: 8.036: [class unloading, 0.0075316 secs]2019-06-05T15:43:52.117+0000: 8.044: [scrub symbol table, 0.0057157 secs]2019-06-05T15:43:52.123+0000: 8.050: [scrub string table, 0.0005016 secs][1 CMS-remark: 4195K(843776K)] 56761K(1028096K), 0.0185717 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
2019-06-05T15:43:52.124+0000: 8.051: [CMS-concurrent-sweep-start]
2019-06-05T15:43:52.126+0000: 8.053: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T15:43:52.126+0000: 8.053: [CMS-concurrent-reset-start]
2019-06-05T15:43:52.145+0000: 8.072: [CMS-concurrent-reset: 0.018/0.018 secs] [Times: user=0.01 sys=0.02, real=0.02 secs] 
Heap
 par new generation   total 184320K, used 97303K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  51% used [0x00000000c0000000, 0x00000000c51f71c8, 0x00000000ca000000)
  from space 20480K,  65% used [0x00000000ca000000, 0x00000000cad0eaf0, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4189K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 31437K, capacity 31780K, committed 32076K, reserved 1077248K
  class space    used 3647K, capacity 3764K, committed 3832K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906051606 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12266924k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/gc.log-201906051616 <==
2019-06-05T16:16:56.405+0000: 2.856: [CMS-concurrent-mark: 0.015/0.015 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] 
2019-06-05T16:16:56.405+0000: 2.856: [CMS-concurrent-preclean-start]
2019-06-05T16:16:56.407+0000: 2.857: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.01 sys=0.01, real=0.00 secs] 
2019-06-05T16:16:56.407+0000: 2.857: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T16:17:01.433+0000: 7.884: [CMS-concurrent-abortable-preclean: 1.041/5.027 secs] [Times: user=2.21 sys=0.03, real=5.03 secs] 
2019-06-05T16:17:01.434+0000: 7.884: [GC (CMS Final Remark) [YG occupancy: 46702 K (184320 K)]2019-06-05T16:17:01.434+0000: 7.884: [Rescan (parallel) , 0.0061621 secs]2019-06-05T16:17:01.440+0000: 7.890: [weak refs processing, 0.0000280 secs]2019-06-05T16:17:01.440+0000: 7.890: [class unloading, 0.0042890 secs]2019-06-05T16:17:01.444+0000: 7.895: [scrub symbol table, 0.0049889 secs]2019-06-05T16:17:01.449+0000: 7.900: [scrub string table, 0.0005608 secs][1 CMS-remark: 4313K(843776K)] 51016K(1028096K), 0.0166945 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2019-06-05T16:17:01.451+0000: 7.901: [CMS-concurrent-sweep-start]
2019-06-05T16:17:01.453+0000: 7.903: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T16:17:01.453+0000: 7.903: [CMS-concurrent-reset-start]
2019-06-05T16:17:01.472+0000: 7.922: [CMS-concurrent-reset: 0.019/0.019 secs] [Times: user=0.01 sys=0.02, real=0.02 secs] 
2019-06-05T16:34:05.956+0000: 1032.406: [GC (Allocation Failure) 2019-06-05T16:34:05.956+0000: 1032.406: [ParNew: 175251K->7507K(184320K), 0.0173576 secs] 179559K->16229K(1028096K), 0.0174691 secs] [Times: user=0.06 sys=0.00, real=0.02 secs] 
2019-06-05T17:13:10.250+0000: 3376.700: [GC (Allocation Failure) 2019-06-05T17:13:10.250+0000: 3376.700: [ParNew: 171347K->1975K(184320K), 0.0049155 secs] 180069K->10697K(1028096K), 0.0050419 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T17:53:55.245+0000: 5821.696: [GC (Allocation Failure) 2019-06-05T17:53:55.245+0000: 5821.696: [ParNew: 165815K->2371K(184320K), 0.0072549 secs] 174537K->11094K(1028096K), 0.0073848 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2019-06-05T18:37:30.249+0000: 8436.699: [GC (Allocation Failure) 2019-06-05T18:37:30.249+0000: 8436.700: [ParNew: 166211K->2123K(184320K), 0.0048336 secs] 174934K->10845K(1028096K), 0.0049468 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-05T19:17:47.200+0000: 10853.650: [GC (Allocation Failure) 2019-06-05T19:17:47.200+0000: 10853.651: [ParNew: 165963K->2827K(184320K), 0.0062859 secs] 174685K->11549K(1028096K), 0.0064075 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T20:02:10.249+0000: 13516.700: [GC (Allocation Failure) 2019-06-05T20:02:10.249+0000: 13516.700: [ParNew: 166667K->2397K(184320K), 0.0048457 secs] 175389K->11119K(1028096K), 0.0049566 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-05T20:47:08.459+0000: 16214.910: [GC (Allocation Failure) 2019-06-05T20:47:08.460+0000: 16214.910: [ParNew: 166237K->1512K(184320K), 0.0059413 secs] 174959K->11152K(1028096K), 0.0060330 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-05T21:29:48.050+0000: 18774.500: [GC (Allocation Failure) 2019-06-05T21:29:48.050+0000: 18774.500: [ParNew: 165352K->770K(184320K), 0.0063862 secs] 174992K->10454K(1028096K), 0.0064844 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-05T22:16:55.682+0000: 21602.132: [GC (Allocation Failure) 2019-06-05T22:16:55.682+0000: 21602.132: [ParNew: 164610K->698K(184320K), 0.0047922 secs] 174294K->10418K(1028096K), 0.0048749 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-05T23:01:59.417+0000: 24305.868: [GC (Allocation Failure) 2019-06-05T23:01:59.418+0000: 24305.868: [ParNew: 164538K->693K(184320K), 0.0043656 secs] 174258K->10429K(1028096K), 0.0044703 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] 
2019-06-05T23:47:17.894+0000: 27024.345: [GC (Allocation Failure) 2019-06-05T23:47:17.894+0000: 27024.345: [ParNew: 164533K->730K(184320K), 0.0045424 secs] 174269K->10503K(1028096K), 0.0046273 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-06T00:33:35.245+0000: 29801.695: [GC (Allocation Failure) 2019-06-06T00:33:35.245+0000: 29801.695: [ParNew: 164570K->723K(184320K), 0.0061116 secs] 174343K->10505K(1028096K), 0.0062001 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2019-06-06T01:19:19.077+0000: 32545.527: [GC (Allocation Failure) 2019-06-06T01:19:19.077+0000: 32545.527: [ParNew: 164563K->777K(184320K), 0.0041349 secs] 174345K->10576K(1028096K), 0.0043211 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-06T02:05:08.439+0000: 35294.889: [GC (Allocation Failure) 2019-06-06T02:05:08.439+0000: 35294.890: [ParNew: 164617K->717K(184320K), 0.0049936 secs] 174416K->10530K(1028096K), 0.0050954 secs] [Times: user=0.02 sys=0.01, real=0.01 secs] 
2019-06-06T02:50:09.398+0000: 37995.848: [GC (Allocation Failure) 2019-06-06T02:50:09.398+0000: 37995.849: [ParNew: 164557K->783K(184320K), 0.0042525 secs] 174370K->10613K(1028096K), 0.0043446 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2019-06-06T03:32:58.056+0000: 40564.506: [GC (Allocation Failure) 2019-06-06T03:32:58.056+0000: 40564.506: [ParNew: 164623K->806K(184320K), 0.0042673 secs] 174453K->10638K(1028096K), 0.0043963 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T04:19:29.506+0000: 43355.956: [GC (Allocation Failure) 2019-06-06T04:19:29.506+0000: 43355.956: [ParNew: 164646K->638K(184320K), 0.0041052 secs] 174478K->10475K(1028096K), 0.0042224 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T05:06:15.252+0000: 46161.703: [GC (Allocation Failure) 2019-06-06T05:06:15.252+0000: 46161.703: [ParNew: 164478K->805K(184320K), 0.0046714 secs] 174315K->10645K(1028096K), 0.0048050 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-06T05:51:58.247+0000: 48904.697: [GC (Allocation Failure) 2019-06-06T05:51:58.247+0000: 48904.697: [ParNew: 164645K->560K(184320K), 0.0042094 secs] 174485K->10465K(1028096K), 0.0043113 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2019-06-06T06:38:31.021+0000: 51697.471: [GC (Allocation Failure) 2019-06-06T06:38:31.021+0000: 51697.471: [ParNew: 164400K->596K(184320K), 0.0041142 secs] 174305K->10502K(1028096K), 0.0041885 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2019-06-06T07:23:58.145+0000: 54424.595: [GC (Allocation Failure) 2019-06-06T07:23:58.145+0000: 54424.595: [ParNew: 164436K->615K(184320K), 0.0043093 secs] 174342K->10526K(1028096K), 0.0044370 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-06-06T08:08:19.869+0000: 57086.320: [GC (GCLocker Initiated GC) 2019-06-06T08:08:19.869+0000: 57086.320: [ParNew: 164455K->625K(184320K), 0.0042283 secs] 174366K->10536K(1028096K), 0.0043188 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
Heap
 par new generation   total 184320K, used 801K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,   0% used [0x00000000c0000000, 0x00000000c002c160, 0x00000000ca000000)
  from space 20480K,   3% used [0x00000000ca000000, 0x00000000ca09c4f0, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 9911K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 36016K, capacity 36442K, committed 36712K, reserved 1081344K
  class space    used 3854K, capacity 3975K, committed 4012K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906060932 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12238728k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-dev-hdfs-restore-clstr-w1.out.2 <==
Error opening zip file or JAR manifest missing : /home/sshhdfsuser/jmx_prometheus_javaagent-0.11.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument
ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 55973
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201906051550 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12275184k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2019-06-05T15:50:27.235+0000: 1.749: [GC (Allocation Failure) 2019-06-05T15:50:27.235+0000: 1.749: [ParNew: 163840K->13453K(184320K), 0.0169553 secs] 163840K->13453K(1028096K), 0.0170754 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
2019-06-05T15:50:29.253+0000: 3.766: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 134943K(1028096K), 0.0100265 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2019-06-05T15:50:29.263+0000: 3.776: [CMS-concurrent-mark-start]
2019-06-05T15:50:29.272+0000: 3.785: [CMS-concurrent-mark: 0.009/0.009 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] 
2019-06-05T15:50:29.272+0000: 3.785: [CMS-concurrent-preclean-start]
2019-06-05T15:50:29.274+0000: 3.787: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T15:50:29.274+0000: 3.787: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 2019-06-05T15:50:34.311+0000: 8.824: [CMS-concurrent-abortable-preclean: 1.291/5.037 secs] [Times: user=1.31 sys=0.00, real=5.04 secs] 
2019-06-05T15:50:34.311+0000: 8.824: [GC (CMS Final Remark) [YG occupancy: 138220 K (184320 K)]2019-06-05T15:50:34.311+0000: 8.824: [Rescan (parallel) , 0.0103779 secs]2019-06-05T15:50:34.321+0000: 8.834: [weak refs processing, 0.0000557 secs]2019-06-05T15:50:34.321+0000: 8.835: [class unloading, 0.0034842 secs]2019-06-05T15:50:34.325+0000: 8.838: [scrub symbol table, 0.0040736 secs]2019-06-05T15:50:34.329+0000: 8.842: [scrub string table, 0.0003243 secs][1 CMS-remark: 0K(843776K)] 138220K(1028096K), 0.0189857 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2019-06-05T15:50:34.330+0000: 8.843: [CMS-concurrent-sweep-start]
2019-06-05T15:50:34.330+0000: 8.843: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-06-05T15:50:34.330+0000: 8.843: [CMS-concurrent-reset-start]
2019-06-05T15:50:34.342+0000: 8.855: [CMS-concurrent-reset: 0.012/0.012 secs] [Times: user=0.00 sys=0.01, real=0.02 secs] 
2019-06-05T15:52:57.235+0000: 151.748: [GC (Allocation Failure) 2019-06-05T15:52:57.235+0000: 151.748: [ParNew: 177293K->13367K(184320K), 0.0526413 secs] 177293K->17968K(1028096K), 0.0527567 secs] [Times: user=0.11 sys=0.02, real=0.05 secs] 
Heap
 par new generation   total 184320K, used 42647K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
  eden space 163840K,  17% used [0x00000000c0000000, 0x00000000c1c97d90, 0x00000000ca000000)
  from space 20480K,  65% used [0x00000000ca000000, 0x00000000cad0dfa0, 0x00000000cb400000)
  to   space 20480K,   0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
 concurrent mark-sweep generation total 843776K, used 4600K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 28699K, capacity 29036K, committed 29244K, reserved 1075200K
  class space    used 3366K, capacity 3471K, committed 3528K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201906051608 <==
OpenJDK 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 23 2018 15:56:20 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-28)
Memory: 4k page, physical 14350480k(12257596k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

Command failed after 1 tries