Member since
04-06-2018
7
Posts
0
Kudos Received
0
Solutions
04-26-2018
06:14 PM
#Hortonworks #Community Nobody helped me, but I found the solution and I think they should add it to the manual of Rolling back namenodeha in the section #modify_hdfs_configurations since the manual is not clear in some things. The next command to get hdfs configurations. # /var/lib/ambari-server/resources/scripts/configs.py -u <AMBARI_USER> -p <AMBARI_PW> -port <AMBARI_PORT> get localhost <CLUSTER_NAME> hdfs-site Before you start change the parameters with yours, the commands was in my case. If you have the next line, delete it since points to namenodeha. "dfs.internal.nameservices"
and you delete with the next command. # /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 delete localhost BigDataCenter hdfs-site dfs.internal.nameservices
Now you find this lines and delete it, if you find the same lines but with "http" also delete it. "dfs.namenode.https-address.namenodeha.nn1" "dfs.namenode.https-address.namenodeha.nn2" # /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 delete localhost BigDataCenter hdfs-site dfs.namenode.https-address.namenodeha.nn1 # /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 delete localhost BigDataCenter hdfs-site dfs.namenode.https-address.namenodeha.nn2 You need to add the next lines since instead of pointing to namenodeha, you have to point to the namenode and add the next lines: "dfs.namenode.http-address" "<namenode>:<port>" "dfs.namenode.https-address" "<namenode>:<port>" # /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 set localhost BigDataCenter hdfs-site dfs.namenode.http-address "master.root.com.mx:50070" # /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 set localhost BigDataCenter hdfs-site dfs.namenode.https-address "master.root.com.mx:50470" and now the cluster works. you can start the services, first start hdfs and then the others.
... View more
04-26-2018
12:16 PM
Hi guys !! i want remove the namenode HA and i followed the next tutorial https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/how_to_roll_back_namenode_ha.html i follow steps but when start only the hdfs service, errors appear, maybe is normal because don't delete the another services, but try start each component from hdfs and those are the logs. In my namenode (master), when start namenode service. stderr: /var/lib/ambari-agent/data/errors-2306.txt Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 424, in <module>
NameNode().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 100, in start
upgrade_suspended=params.upgrade_suspended, env=env)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 226, in namenode
create_hdfs_directories()
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 293, in create_hdfs_directories
mode=0777,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 555, in action_create_on_execute
self.action_delayed("create")
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 552, in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name, self)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 272, in action_delayed
main_resource.resource.security_enabled, main_resource.resource.logoutput)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 138, in __init__
security_enabled, run_user)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/namenode_ha_utils.py", line 173, in get_property_for_active_namenode
if INADDR_ANY in value and rpc_key in hdfs_site:
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'dfs.namenode.https-address' was not found in configurations dictionary! stdout: /var/lib/ambari-agent/data/output-2306.txt 2018-04-25 18:52:56,682 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-04-25 18:52:57,006 - Stack Feature Version Info: stack_version=2.6, version=2.6.4.0-91, current_cluster_version=2.6.4.0-91 -> 2.6.4.0-91
2018-04-25 18:52:57,018 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-04-25 18:52:57,020 - Group['livy'] {}
2018-04-25 18:52:57,022 - Group['spark'] {}
2018-04-25 18:52:57,022 - Group['ranger'] {}
2018-04-25 18:52:57,022 - Group['hadoop'] {}
2018-04-25 18:52:57,023 - Group['users'] {}
2018-04-25 18:52:57,023 - Group['knox'] {}
2018-04-25 18:52:57,023 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,024 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,025 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,026 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,027 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,029 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-25 18:52:57,030 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,031 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-25 18:52:57,032 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']}
2018-04-25 18:52:57,033 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-25 18:52:57,034 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,035 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,036 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,037 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-25 18:52:57,038 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,039 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,040 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,041 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,042 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,043 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,044 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,045 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,046 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 18:52:57,047 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-25 18:52:57,049 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-04-25 18:52:57,060 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-04-25 18:52:57,061 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-04-25 18:52:57,062 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-25 18:52:57,063 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-04-25 18:52:57,074 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2018-04-25 18:52:57,075 - Group['hdfs'] {}
2018-04-25 18:52:57,075 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2018-04-25 18:52:57,076 - FS Type:
2018-04-25 18:52:57,076 - Directory['/etc/hadoop'] {'mode': 0755}
2018-04-25 18:52:57,100 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-25 18:52:57,101 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-04-25 18:52:57,123 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-04-25 18:52:57,136 - Skipping Execute[('setenforce', '0')] due to not_if
2018-04-25 18:52:57,136 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-04-25 18:52:57,143 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-04-25 18:52:57,144 - Changing owner for /var/run/hadoop from 1016 to root
2018-04-25 18:52:57,144 - Changing group for /var/run/hadoop from 1003 to root
2018-04-25 18:52:57,145 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-04-25 18:52:57,156 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-04-25 18:52:57,161 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-04-25 18:52:57,179 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-04-25 18:52:57,208 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-25 18:52:57,210 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-04-25 18:52:57,212 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-04-25 18:52:57,223 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2018-04-25 18:52:57,232 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-04-25 18:52:57,650 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-04-25 18:52:57,651 - Stack Feature Version Info: stack_version=2.6, version=2.6.4.0-91, current_cluster_version=2.6.4.0-91 -> 2.6.4.0-91
2018-04-25 18:52:57,680 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-04-25 18:52:57,697 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1}
2018-04-25 18:52:57,739 - checked_call returned (0, '2.6.4.0-91', '')
2018-04-25 18:52:57,749 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2018-04-25 18:52:57,756 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2018-04-25 18:52:57,756 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-04-25 18:52:57,768 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
2018-04-25 18:52:57,768 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 18:52:57,777 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-04-25 18:52:57,787 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2018-04-25 18:52:57,787 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 18:52:57,794 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-04-25 18:52:57,795 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2018-04-25 18:52:57,805 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2018-04-25 18:52:57,805 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 18:52:57,812 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-04-25 18:52:57,822 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2018-04-25 18:52:57,822 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 18:52:57,829 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'configurations': ...}
2018-04-25 18:52:57,839 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2018-04-25 18:52:57,839 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 18:52:57,883 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-04-25 18:52:57,893 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2018-04-25 18:52:57,893 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-04-25 18:52:57,919 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'}
2018-04-25 18:52:57,924 - Directory['/mnt/namenode'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-04-25 18:52:57,924 - Skipping setting up secure ZNode ACL for HFDS as it's supported only for NameNode HA mode.
2018-04-25 18:52:57,928 - Called service start with upgrade_type: None
2018-04-25 18:52:57,928 - HDFS: Setup ranger: command retry not enabled thus skipping if ranger admin is down !
2018-04-25 18:52:57,929 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-client'] {'timeout': 20}
2018-04-25 18:52:57,988 - call returned (0, 'hadoop-client - 2.6.4.0-91')
2018-04-25 18:52:57,992 - Skipping Ranger API calls, as policy cache file exists for hdfs
2018-04-25 18:52:57,992 - If service name for hdfs is not created on Ranger Admin UI, then to re-create it delete policy cache file: /etc/ranger/BigDataCenter_hadoop/policycache/hdfs_BigDataCenter_hadoop.json
2018-04-25 18:52:57,994 - File['/usr/hdp/current/hadoop-client/conf/ranger-security.xml'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-04-25 18:52:57,994 - Writing File['/usr/hdp/current/hadoop-client/conf/ranger-security.xml'] because contents don't match
2018-04-25 18:52:57,995 - Directory['/etc/ranger/BigDataCenter_hadoop'] {'owner': 'hdfs', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-04-25 18:52:57,995 - Directory['/etc/ranger/BigDataCenter_hadoop/policycache'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-04-25 18:52:57,996 - File['/etc/ranger/BigDataCenter_hadoop/policycache/hdfs_BigDataCenter_hadoop.json'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-04-25 18:52:57,996 - XmlConfig['ranger-hdfs-audit.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0744, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2018-04-25 18:52:58,008 - Generating config: /usr/hdp/current/hadoop-client/conf/ranger-hdfs-audit.xml
2018-04-25 18:52:58,008 - File['/usr/hdp/current/hadoop-client/conf/ranger-hdfs-audit.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0744, 'encoding': 'UTF-8'}
2018-04-25 18:52:58,016 - XmlConfig['ranger-hdfs-security.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0744, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2018-04-25 18:52:58,026 - Generating config: /usr/hdp/current/hadoop-client/conf/ranger-hdfs-security.xml
2018-04-25 18:52:58,027 - File['/usr/hdp/current/hadoop-client/conf/ranger-hdfs-security.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0744, 'encoding': 'UTF-8'}
2018-04-25 18:52:58,034 - XmlConfig['ranger-policymgr-ssl.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0744, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2018-04-25 18:52:58,044 - Generating config: /usr/hdp/current/hadoop-client/conf/ranger-policymgr-ssl.xml
2018-04-25 18:52:58,044 - File['/usr/hdp/current/hadoop-client/conf/ranger-policymgr-ssl.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0744, 'encoding': 'UTF-8'}
2018-04-25 18:52:58,050 - Execute[(u'/usr/hdp/2.6.4.0-91/ranger-hdfs-plugin/ranger_credential_helper.py', '-l', u'/usr/hdp/2.6.4.0-91/ranger-hdfs-plugin/install/lib/*', '-f', '/etc/ranger/BigDataCenter_hadoop/cred.jceks', '-k', 'sslKeyStore', '-v', [PROTECTED], '-c', '1')] {'logoutput': True, 'environment': {'JAVA_HOME': u'/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64'}, 'sudo': True}
Using Java:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java
Alias sslKeyStore created successfully!
2018-04-25 18:52:59,275 - Execute[(u'/usr/hdp/2.6.4.0-91/ranger-hdfs-plugin/ranger_credential_helper.py', '-l', u'/usr/hdp/2.6.4.0-91/ranger-hdfs-plugin/install/lib/*', '-f', '/etc/ranger/BigDataCenter_hadoop/cred.jceks', '-k', 'sslTrustStore', '-v', [PROTECTED], '-c', '1')] {'logoutput': True, 'environment': {'JAVA_HOME': u'/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64'}, 'sudo': True}
Using Java:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java
Alias sslTrustStore created successfully!
2018-04-25 18:53:00,367 - File['/etc/ranger/BigDataCenter_hadoop/cred.jceks'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0640}
2018-04-25 18:53:00,371 - File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'), 'group': 'hadoop'}
2018-04-25 18:53:00,372 - call[('ls', u'/mnt/namenode')] {}
2018-04-25 18:53:00,379 - call returned (0, 'current\nin_use.lock\nlost+found')
2018-04-25 18:53:00,379 - Execute['ls /mnt/namenode | wc -l | grep -q ^0] {}
2018-04-25 18:53:00,389 - NameNode will not be formatted since /mnt/namenode exists and contains content
2018-04-25 18:53:00,389 - Options for start command are:
2018-04-25 18:53:00,391 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2018-04-25 18:53:00,392 - Changing owner for /var/run/hadoop from 0 to hdfs
2018-04-25 18:53:00,392 - Changing group for /var/run/hadoop from 0 to hadoop
2018-04-25 18:53:00,393 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2018-04-25 18:53:00,393 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2018-04-25 18:53:00,395 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2018-04-25 18:53:00,425 - Skipping File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] due to not_if
2018-04-25 18:53:00,426 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2018-04-25 18:53:00,455 - Skipping Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode''] due to not_if
2018-04-25 18:53:00,456 - Waiting for this NameNode to leave Safemode due to the following conditions: HA: False, isActive: True, upgradeType: None
2018-04-25 18:53:00,456 - Waiting up to 19 minutes for the NameNode to leave Safemode...
2018-04-25 18:53:00,457 - Execute['/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://master.root.com.mx:8020 -safemode get | grep 'Safe mode is OFF''] {'logoutput': True, 'tries': 115, 'user': 'hdfs', 'try_sleep': 10}
Safe mode is OFF
2018-04-25 18:53:02,786 - HdfsResource['/tmp'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.root.com.mx:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0777}
Command failed after 1 tries The another hdfs components works except the namenode service, but if i start all hdfs, always the datanodes doesn't work and this is the log. thank you so much and sorry for my english stderr: /var/lib/ambari-agent/data/errors-2315.txt Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 177, in <module>
DataNode().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 762, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 64, in start
datanode(action="start")
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py", line 68, in datanode
create_log_dir=True
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 271, in service
Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.out stdout: /var/lib/ambari-agent/data/output-2315.txt 2018-04-25 19:06:00,420 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-04-25 19:06:00,697 - Stack Feature Version Info: stack_version=2.6, version=2.6.4.0-91, current_cluster_version=2.6.4.0-91 -> 2.6.4.0-91
2018-04-25 19:06:00,706 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-04-25 19:06:00,708 - Group['livy'] {}
2018-04-25 19:06:00,709 - Group['spark'] {}
2018-04-25 19:06:00,709 - Group['ranger'] {}
2018-04-25 19:06:00,709 - Group['hadoop'] {}
2018-04-25 19:06:00,709 - Group['users'] {}
2018-04-25 19:06:00,710 - Group['knox'] {}
2018-04-25 19:06:00,710 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,711 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,712 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,712 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,713 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,714 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-25 19:06:00,715 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,716 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-25 19:06:00,716 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']}
2018-04-25 19:06:00,717 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-25 19:06:00,718 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,719 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,720 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,720 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-25 19:06:00,721 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,722 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,723 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,723 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,724 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,725 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,726 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,727 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,727 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-25 19:06:00,728 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-25 19:06:00,730 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-04-25 19:06:00,741 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-04-25 19:06:00,741 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-04-25 19:06:00,742 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-25 19:06:00,743 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-04-25 19:06:00,754 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2018-04-25 19:06:00,754 - Group['hdfs'] {}
2018-04-25 19:06:00,754 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2018-04-25 19:06:00,755 - FS Type:
2018-04-25 19:06:00,755 - Directory['/etc/hadoop'] {'mode': 0755}
2018-04-25 19:06:00,775 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-25 19:06:00,775 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-04-25 19:06:00,794 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-04-25 19:06:00,807 - Skipping Execute[('setenforce', '0')] due to not_if
2018-04-25 19:06:00,808 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-04-25 19:06:00,813 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-04-25 19:06:00,814 - Changing owner for /var/run/hadoop from 1016 to root
2018-04-25 19:06:00,814 - Changing group for /var/run/hadoop from 1003 to root
2018-04-25 19:06:00,815 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-04-25 19:06:00,827 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-04-25 19:06:00,832 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-04-25 19:06:00,851 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-04-25 19:06:00,880 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-25 19:06:00,881 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-04-25 19:06:00,883 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-04-25 19:06:00,895 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2018-04-25 19:06:00,902 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-04-25 19:06:01,306 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-04-25 19:06:01,307 - Stack Feature Version Info: stack_version=2.6, version=2.6.4.0-91, current_cluster_version=2.6.4.0-91 -> 2.6.4.0-91
2018-04-25 19:06:01,336 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-04-25 19:06:01,353 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1}
2018-04-25 19:06:01,442 - checked_call returned (0, '2.6.4.0-91', '')
2018-04-25 19:06:01,457 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf stop datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'only_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2018-04-25 19:06:01,486 - Skipping Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf stop datanode''] due to only_if
2018-04-25 19:06:01,487 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete']}
2018-04-25 19:06:01,489 - Deleting File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid']
2018-04-25 19:06:01,490 - Pid file /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid is empty or does not exist
2018-04-25 19:06:01,495 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2018-04-25 19:06:01,503 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2018-04-25 19:06:01,505 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-04-25 19:06:01,520 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
2018-04-25 19:06:01,521 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 19:06:01,534 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-04-25 19:06:01,547 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2018-04-25 19:06:01,547 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 19:06:01,556 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-04-25 19:06:01,557 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2018-04-25 19:06:01,570 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2018-04-25 19:06:01,571 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 19:06:01,580 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-04-25 19:06:01,592 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2018-04-25 19:06:01,593 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 19:06:01,602 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'configurations': ...}
2018-04-25 19:06:01,615 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2018-04-25 19:06:01,616 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-04-25 19:06:01,664 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-04-25 19:06:01,673 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2018-04-25 19:06:01,674 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-04-25 19:06:01,699 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'}
2018-04-25 19:06:01,701 - Directory['/var/lib/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'group': 'hadoop', 'mode': 0751}
2018-04-25 19:06:01,701 - Directory['/var/lib/ambari-agent/data/datanode'] {'create_parents': True, 'mode': 0755}
2018-04-25 19:06:01,709 - Host contains mounts: ['/sys', '/proc', '/dev', '/sys/kernel/security', '/dev/shm', '/dev/pts', '/run', '/sys/fs/cgroup', '/sys/fs/cgroup/systemd', '/sys/fs/pstore', '/sys/fs/cgroup/blkio', '/sys/fs/cgroup/cpuset', '/sys/fs/cgroup/perf_event', '/sys/fs/cgroup/net_cls,net_prio', '/sys/fs/cgroup/memory', '/sys/fs/cgroup/freezer', '/sys/fs/cgroup/devices', '/sys/fs/cgroup/pids', '/sys/fs/cgroup/cpu,cpuacct', '/sys/fs/cgroup/hugetlb', '/sys/kernel/config', '/', '/proc/sys/fs/binfmt_misc', '/sys/kernel/debug', '/dev/hugepages', '/dev/mqueue', '/proc/fs/nfsd', '/mnt/datanode', '/boot', '/var/lib/nfs/rpc_pipefs', '/run/user/42', '/run/user/1016'].
2018-04-25 19:06:01,709 - Mount point for directory /mnt/datanode is /mnt/datanode
2018-04-25 19:06:01,709 - Mount point for directory /mnt/datanode is /mnt/datanode
2018-04-25 19:06:01,710 - Forcefully ensuring existence and permissions of the directory: /mnt/datanode
2018-04-25 19:06:01,710 - Directory['/mnt/datanode'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'ignore_failures': True, 'mode': 0755, 'owner': 'hdfs'}
2018-04-25 19:06:01,710 - Changing permission for /mnt/datanode from 750 to 755
2018-04-25 19:06:01,718 - Host contains mounts: ['/sys', '/proc', '/dev', '/sys/kernel/security', '/dev/shm', '/dev/pts', '/run', '/sys/fs/cgroup', '/sys/fs/cgroup/systemd', '/sys/fs/pstore', '/sys/fs/cgroup/blkio', '/sys/fs/cgroup/cpuset', '/sys/fs/cgroup/perf_event', '/sys/fs/cgroup/net_cls,net_prio', '/sys/fs/cgroup/memory', '/sys/fs/cgroup/freezer', '/sys/fs/cgroup/devices', '/sys/fs/cgroup/pids', '/sys/fs/cgroup/cpu,cpuacct', '/sys/fs/cgroup/hugetlb', '/sys/kernel/config', '/', '/proc/sys/fs/binfmt_misc', '/sys/kernel/debug', '/dev/hugepages', '/dev/mqueue', '/proc/fs/nfsd', '/mnt/datanode', '/boot', '/var/lib/nfs/rpc_pipefs', '/run/user/42', '/run/user/1016'].
2018-04-25 19:06:01,719 - Mount point for directory /mnt/datanode is /mnt/datanode
2018-04-25 19:06:01,719 - File['/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist'] {'content': '\n# This file keeps track of the last known mount-point for each dir.\n# It is safe to delete, since it will get regenerated the next time that the component of the service starts.\n# However, it is not advised to delete this file since Ambari may\n# re-create a dir that used to be mounted on a drive but is now mounted on the root.\n# Comments begin with a hash (#) symbol\n# dir,mount_point\n/mnt/datanode1,/mnt/datanode1\n/mnt/datanode,/mnt/datanode\n/mnt/datanode3,/\n/mnt/datanode2,/\n', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-04-25 19:06:01,720 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2018-04-25 19:06:01,721 - Changing owner for /var/run/hadoop from 0 to hdfs
2018-04-25 19:06:01,721 - Changing group for /var/run/hadoop from 0 to hadoop
2018-04-25 19:06:01,721 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2018-04-25 19:06:01,721 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2018-04-25 19:06:01,722 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2018-04-25 19:06:01,730 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'}
2018-04-25 19:06:05,960 - Execute['find /var/log/hadoop/hdfs -maxdepth 1 -type f -name '*' -exec echo '==> {} <==' \; -exec tail -n 40 {} \;'] {'logoutput': True, 'ignore_failures': True, 'user': 'hdfs'}
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.9 <==
2018-04-04 17:59:37,289 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376496_1635682 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376496 for deletion
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376489_1635675 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376489
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376497_1635683 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376497 for deletion
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376490_1635676 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376490
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376498_1635684 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376498 for deletion
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376491_1635677 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376491
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376499_1635685 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376499 for deletion
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376492_1635678 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376492
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376500_1635686 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376500 for deletion
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376493_1635679 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376493
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376501_1635687 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376501 for deletion
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376494_1635680 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376494
2018-04-04 17:59:37,290 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376502_1635688 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376502 for deletion
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376495_1635681 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376495
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376503_1635689 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376503 for deletion
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376496_1635682 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376496
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376504_1635690 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376504 for deletion
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376497_1635683 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376497
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376505_1635691 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376505 for deletion
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376498_1635684 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376498
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075376506_1635692 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376506 for deletion
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376499_1635685 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376499
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376500_1635686 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376500
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376501_1635687 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376501
2018-04-04 17:59:37,291 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376502_1635688 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376502
2018-04-04 17:59:37,292 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376503_1635689 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376503
2018-04-04 17:59:37,292 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376504_1635690 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376504
2018-04-04 17:59:37,292 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376505_1635691 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376505
2018-04-04 17:59:37,292 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075376506_1635692 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir24/subdir241/blk_1075376506
2018-04-04 17:59:37,534 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075376513_1635699 src: /10.1.5.21:59534 dest: /10.1.5.20:50010
2018-04-04 17:59:37,536 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:59534, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075376513_1635699, duration: 961475
2018-04-04 17:59:37,536 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075376513_1635699, type=LAST_IN_PIPELINE terminating
2018-04-04 17:59:37,637 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075376514_1635700 src: /10.1.5.21:59538 dest: /10.1.5.20:50010
2018-04-04 17:59:37,639 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:59538, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075376514_1635700, duration: 908071
2018-04-04 17:59:37,639 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075376514_1635700, type=LAST_IN_PIPELINE terminating
2018-04-04 17:59:37,731 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075376515_1635701 src: /10.1.5.21:59542 dest: /10.1.5.20:50010
2018-04-04 17:59:37,732 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:59542, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075376515_1635701, duration: 912956
2018-04-04 17:59:37,733 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075376515_1635701, type=LAST_IN_PIPELINE terminating
2018-04-04 17:59:38,269 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075376516_1635702 src: /10.1.5.21:59546 dest: /10.1.5.20:50010
2018-04-04 17:59:38,272 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:59546, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075376516_1635702, duration: 1115848
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.out <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804020918 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(7661276k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T09:18:30.863-0500: 1.986: [GC (Allocation Failure) 2018-04-02T09:18:30.863-0500: 1.986: [ParNew: 163840K->13895K(184320K), 0.0249692 secs] 163840K->13895K(1028096K), 0.0251486 secs] [Times: user=0.04 sys=0.01, real=0.02 secs]
2018-04-02T09:18:32.888-0500: 4.011: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 118003K(1028096K), 0.0150229 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-02T09:18:32.903-0500: 4.026: [CMS-concurrent-mark-start]
2018-04-02T09:18:32.907-0500: 4.030: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T09:18:32.908-0500: 4.030: [CMS-concurrent-preclean-start]
2018-04-02T09:18:32.909-0500: 4.031: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T09:18:32.909-0500: 4.031: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2018-04-02T09:18:38.003-0500: 9.125: [CMS-concurrent-abortable-preclean: 1.914/5.094 secs] [Times: user=1.95 sys=0.01, real=5.10 secs]
2018-04-02T09:18:38.003-0500: 9.126: [GC (CMS Final Remark) [YG occupancy: 121280 K (184320 K)]2018-04-02T09:18:38.003-0500: 9.126: [Rescan (parallel) , 0.0134580 secs]2018-04-02T09:18:38.017-0500: 9.139: [weak refs processing, 0.0000451 secs]2018-04-02T09:18:38.017-0500: 9.139: [class unloading, 0.0057625 secs]2018-04-02T09:18:38.023-0500: 9.145: [scrub symbol table, 0.0052773 secs]2018-04-02T09:18:38.028-0500: 9.150: [scrub string table, 0.0004422 secs][1 CMS-remark: 0K(843776K)] 121280K(1028096K), 0.0259444 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-02T09:18:38.029-0500: 9.152: [CMS-concurrent-sweep-start]
2018-04-02T09:18:38.029-0500: 9.152: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T09:18:38.029-0500: 9.152: [CMS-concurrent-reset-start]
2018-04-02T09:18:38.037-0500: 9.159: [CMS-concurrent-reset: 0.007/0.007 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2018-04-02T09:23:37.086-0500: 308.208: [GC (Allocation Failure) 2018-04-02T09:23:37.086-0500: 308.208: [ParNew: 177735K->12829K(184320K), 0.0371601 secs] 177735K->17228K(1028096K), 0.0373112 secs] [Times: user=0.09 sys=0.02, real=0.04 secs]
2018-04-02T09:47:01.665-0500: 1712.787: [GC (Allocation Failure) 2018-04-02T09:47:01.665-0500: 1712.787: [ParNew: 176669K->7366K(184320K), 0.0218355 secs] 181068K->17231K(1028096K), 0.0220506 secs] [Times: user=0.07 sys=0.01, real=0.02 secs]
2018-04-02T10:11:09.148-0500: 3160.270: [GC (Allocation Failure) 2018-04-02T10:11:09.148-0500: 3160.270: [ParNew: 171206K->5720K(184320K), 0.0136411 secs] 181071K->15584K(1028096K), 0.0138381 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
2018-04-02T10:52:35.357-0500: 5646.479: [GC (Allocation Failure) 2018-04-02T10:52:35.357-0500: 5646.479: [ParNew: 169560K->2443K(184320K), 0.0110320 secs] 179424K->12307K(1028096K), 0.0112380 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 71545K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 42% used [0x00000000c0000000, 0x00000000c437b8b0, 0x00000000ca000000)
from space 20480K, 11% used [0x00000000cb400000, 0x00000000cb662c80, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 9864K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 37117K, capacity 37882K, committed 38052K, reserved 1083392K
class space used 4028K, capacity 4243K, committed 4296K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804061611 <==
2018-04-08T05:29:29.123-0500: 134292.042: [GC (Allocation Failure) 2018-04-08T05:29:29.123-0500: 134292.042: [ParNew: 105294K->293K(118016K), 0.0048539 secs] 126248K->21254K(1035520K), 0.0049830 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-08T07:04:29.313-0500: 139992.233: [GC (Allocation Failure) 2018-04-08T07:04:29.313-0500: 139992.233: [ParNew: 105253K->275K(118016K), 0.0049585 secs] 126214K->21236K(1035520K), 0.0050862 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]
2018-04-08T08:39:22.579-0500: 145685.498: [GC (Allocation Failure) 2018-04-08T08:39:22.579-0500: 145685.499: [ParNew: 105235K->258K(118016K), 0.0052718 secs] 126196K->21221K(1035520K), 0.0054087 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-08T10:25:30.227-0500: 152053.147: [GC (Allocation Failure) 2018-04-08T10:25:30.227-0500: 152053.147: [ParNew: 105218K->285K(118016K), 0.0053775 secs] 126181K->21252K(1035520K), 0.0055040 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-08T12:22:30.448-0500: 159073.367: [GC (Allocation Failure) 2018-04-08T12:22:30.448-0500: 159073.367: [ParNew: 105245K->321K(118016K), 0.0053325 secs] 126212K->21288K(1035520K), 0.0054636 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-08T13:55:30.631-0500: 164653.551: [GC (Allocation Failure) 2018-04-08T13:55:30.631-0500: 164653.551: [ParNew: 105281K->265K(118016K), 0.0048938 secs] 126248K->21232K(1035520K), 0.0050203 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-08T15:52:22.741-0500: 171665.661: [GC (Allocation Failure) 2018-04-08T15:52:22.741-0500: 171665.661: [ParNew: 105225K->294K(118016K), 0.0045041 secs] 126192K->21265K(1035520K), 0.0046345 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-08T17:18:22.602-0500: 176825.522: [GC (Allocation Failure) 2018-04-08T17:18:22.602-0500: 176825.522: [ParNew: 105254K->245K(118016K), 0.0052100 secs] 126225K->21216K(1035520K), 0.0053352 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-08T19:15:22.678-0500: 183845.597: [GC (Allocation Failure) 2018-04-08T19:15:22.678-0500: 183845.597: [ParNew: 105205K->307K(118016K), 0.0051979 secs] 126176K->21278K(1035520K), 0.0053215 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-08T21:12:06.865-0500: 190849.784: [GC (Allocation Failure) 2018-04-08T21:12:06.865-0500: 190849.784: [ParNew: 105267K->327K(118016K), 0.0059338 secs] 126238K->21301K(1035520K), 0.0060947 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-08T22:37:22.582-0500: 195965.501: [GC (Allocation Failure) 2018-04-08T22:37:22.582-0500: 195965.501: [ParNew: 105287K->256K(118016K), 0.0030787 secs] 126261K->21230K(1035520K), 0.0031614 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-09T00:34:22.589-0500: 202985.508: [GC (Allocation Failure) 2018-04-09T00:34:22.589-0500: 202985.508: [ParNew: 105216K->305K(118016K), 0.0055688 secs] 126190K->21290K(1035520K), 0.0057180 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-09T02:09:32.872-0500: 208695.791: [GC (Allocation Failure) 2018-04-09T02:09:32.872-0500: 208695.791: [ParNew: 105265K->242K(118016K), 0.0046086 secs] 126250K->21228K(1035520K), 0.0047614 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-09T03:44:22.569-0500: 214385.488: [GC (Allocation Failure) 2018-04-09T03:44:22.569-0500: 214385.488: [ParNew: 105202K->239K(118016K), 0.0050033 secs] 126188K->21226K(1035520K), 0.0051255 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-09T05:10:33.739-0500: 219556.658: [GC (Allocation Failure) 2018-04-09T05:10:33.739-0500: 219556.659: [ParNew: 105199K->254K(118016K), 0.0047484 secs] 126186K->21241K(1035520K), 0.0049039 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-09T06:45:33.930-0500: 225256.849: [GC (Allocation Failure) 2018-04-09T06:45:33.930-0500: 225256.849: [ParNew: 105214K->273K(118016K), 0.0042178 secs] 126201K->21260K(1035520K), 0.0043946 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-09T08:20:34.117-0500: 230957.036: [GC (Allocation Failure) 2018-04-09T08:20:34.117-0500: 230957.036: [ParNew: 105233K->276K(118016K), 0.0047991 secs] 126220K->21268K(1035520K), 0.0049569 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T09:55:34.305-0500: 236657.224: [GC (Allocation Failure) 2018-04-09T09:55:34.305-0500: 236657.224: [ParNew: 105236K->257K(118016K), 0.0047490 secs] 126228K->21252K(1035520K), 0.0048861 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T11:21:34.815-0500: 241817.734: [GC (Allocation Failure) 2018-04-09T11:21:34.815-0500: 241817.734: [ParNew: 105217K->258K(118016K), 0.0033565 secs] 126212K->21253K(1035520K), 0.0034736 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T12:56:12.607-0500: 247495.526: [GC (Allocation Failure) 2018-04-09T12:56:12.607-0500: 247495.526: [ParNew: 105218K->252K(118016K), 0.0048030 secs] 126213K->21247K(1035520K), 0.0049357 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T14:31:22.543-0500: 253205.462: [GC (Allocation Failure) 2018-04-09T14:31:22.543-0500: 253205.462: [ParNew: 105212K->243K(118016K), 0.0056638 secs] 126207K->21240K(1035520K), 0.0058194 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T16:16:35.937-0500: 259518.856: [GC (Allocation Failure) 2018-04-09T16:16:35.937-0500: 259518.856: [ParNew: 105203K->292K(118016K), 0.0049312 secs] 126200K->21289K(1035520K), 0.0050947 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-09T18:13:36.182-0500: 266539.101: [GC (Allocation Failure) 2018-04-09T18:13:36.182-0500: 266539.101: [ParNew: 105252K->320K(118016K), 0.0049212 secs] 126249K->21322K(1035520K), 0.0050599 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-09T19:48:36.369-0500: 272239.288: [GC (Allocation Failure) 2018-04-09T19:48:36.369-0500: 272239.288: [ParNew: 105280K->273K(118016K), 0.0042952 secs] 126282K->21275K(1035520K), 0.0044131 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-09T21:23:22.560-0500: 277925.479: [GC (Allocation Failure) 2018-04-09T21:23:22.560-0500: 277925.479: [ParNew: 105233K->253K(118016K), 0.0048211 secs] 126235K->21258K(1035520K), 0.0049297 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-09T22:29:03.638-0500: 281866.558: [GC (Allocation Failure) 2018-04-09T22:29:03.638-0500: 281866.558: [ParNew: 105213K->244K(118016K), 0.0049415 secs] 126218K->21249K(1035520K), 0.0050559 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-10T00:25:37.431-0500: 288860.350: [GC (Allocation Failure) 2018-04-10T00:25:37.431-0500: 288860.350: [ParNew: 105204K->345K(118016K), 0.0039306 secs] 126209K->21350K(1035520K), 0.0040568 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-10T02:22:37.658-0500: 295880.577: [GC (Allocation Failure) 2018-04-10T02:22:37.658-0500: 295880.577: [ParNew: 105305K->346K(118016K), 0.0051797 secs] 126310K->21354K(1035520K), 0.0053383 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-10T04:12:38.185-0500: 302481.105: [GC (Allocation Failure) 2018-04-10T04:12:38.185-0500: 302481.105: [ParNew: 105306K->373K(118016K), 0.0051746 secs] 126314K->21383K(1035520K), 0.0052676 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
2018-04-10T06:05:22.545-0500: 309245.464: [GC (Allocation Failure) 2018-04-10T06:05:22.545-0500: 309245.464: [ParNew: 105333K->317K(118016K), 0.0063324 secs] 126343K->21326K(1035520K), 0.0064620 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-10T08:02:22.576-0500: 316265.495: [GC (Allocation Failure) 2018-04-10T08:02:22.576-0500: 316265.495: [ParNew: 105277K->333K(118016K), 0.0061033 secs] 126286K->21343K(1035520K), 0.0062845 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-10T09:37:38.985-0500: 321981.905: [GC (Allocation Failure) 2018-04-10T09:37:38.985-0500: 321981.905: [ParNew: 105293K->299K(118016K), 0.0048579 secs] 126303K->21330K(1035520K), 0.0049947 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
Heap
par new generation total 118016K, used 34159K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 32% used [0x00000000c0000000, 0x00000000c2110f30, 0x00000000c6680000)
from space 13056K, 2% used [0x00000000c7340000, 0x00000000c738ad20, 0x00000000c8000000)
to space 13056K, 0% used [0x00000000c6680000, 0x00000000c6680000, 0x00000000c7340000)
concurrent mark-sweep generation total 917504K, used 21031K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 26485K, capacity 26720K, committed 27212K, reserved 1073152K
class space used 2745K, capacity 2842K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804251738 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(8878532k free), swap 4194300k(4141584k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T17:38:31.798-0500: 2.097: [GC (Allocation Failure) 2018-04-25T17:38:31.798-0500: 2.097: [ParNew: 104960K->11195K(118016K), 0.0156288 secs] 104960K->11195K(1035520K), 0.0157688 secs] [Times: user=0.06 sys=0.03, real=0.01 secs]
2018-04-25T17:39:33.456-0500: 63.754: [GC (Allocation Failure) 2018-04-25T17:39:33.456-0500: 63.754: [ParNew: 116155K->12826K(118016K), 0.0881466 secs] 116155K->24940K(1035520K), 0.0882667 secs] [Times: user=0.56 sys=0.02, real=0.08 secs]
2018-04-25T17:39:33.546-0500: 63.844: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12114K(917504K)] 25520K(1035520K), 0.0123400 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T17:39:33.558-0500: 63.857: [CMS-concurrent-mark-start]
2018-04-25T17:39:33.569-0500: 63.868: [CMS-concurrent-mark: 0.011/0.011 secs] [Times: user=0.08 sys=0.00, real=0.01 secs]
2018-04-25T17:39:33.570-0500: 63.868: [CMS-concurrent-preclean-start]
2018-04-25T17:39:33.573-0500: 63.872: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-25T17:39:33.573-0500: 63.872: [GC (CMS Final Remark) [YG occupancy: 13405 K (118016 K)]2018-04-25T17:39:33.573-0500: 63.872: [Rescan (parallel) , 0.0115248 secs]2018-04-25T17:39:33.585-0500: 63.883: [weak refs processing, 0.0000479 secs]2018-04-25T17:39:33.585-0500: 63.883: [class unloading, 0.0037478 secs]2018-04-25T17:39:33.589-0500: 63.887: [scrub symbol table, 0.0047269 secs]2018-04-25T17:39:33.593-0500: 63.892: [scrub string table, 0.0005772 secs][1 CMS-remark: 12114K(917504K)] 25520K(1035520K), 0.0216421 secs] [Times: user=0.11 sys=0.00, real=0.03 secs]
2018-04-25T17:39:33.595-0500: 63.894: [CMS-concurrent-sweep-start]
2018-04-25T17:39:33.599-0500: 63.897: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-25T17:39:33.599-0500: 63.897: [CMS-concurrent-reset-start]
2018-04-25T17:39:33.603-0500: 63.902: [CMS-concurrent-reset: 0.005/0.005 secs] [Times: user=0.02 sys=0.01, real=0.00 secs]
Heap
par new generation total 118016K, used 82389K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 66% used [0x00000000c0000000, 0x00000000c43eeb98, 0x00000000c6680000)
from space 13056K, 98% used [0x00000000c6680000, 0x00000000c7306a30, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 12109K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 24252K, capacity 24484K, committed 24908K, reserved 1071104K
class space used 2668K, capacity 2779K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-journalnode-nodo1.root.com.mx.log <==
2018-04-25 15:07:21,289 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020151160 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151160-0000000000020151167
2018-04-25 15:09:21,640 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020151168 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151168-0000000000020151175
2018-04-25 15:11:21,969 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020151176 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151176-0000000000020151177
2018-04-25 15:13:22,397 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020151178 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151178-0000000000020151362
2018-04-25 15:15:22,742 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020151363 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151363-0000000000020151520
2018-04-25 15:15:23,198 INFO namenode.TransferFsImage (TransferFsImage.java:copyFileToStream(395)) - Sending fileName: /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151363-0000000000020151520, fileSize: 24458. Sent total: 24458 bytes. Size of last segment intended to send: -1 bytes.
2018-04-25 15:17:23,310 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020151521 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151521-0000000000020151984
2018-04-25 15:19:23,697 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020151985 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151985-0000000000020151998
2018-04-25 15:21:24,272 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020151999 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020151999-0000000000020152006
2018-04-25 15:23:24,622 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020152007 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020152007-0000000000020152029
2018-04-25 15:25:25,105 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020152030 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020152030-0000000000020152044
2018-04-25 15:27:25,438 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020152045 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020152045-0000000000020152072
2018-04-25 15:29:26,275 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020152073 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020152073-0000000000020152843
2018-04-25 15:31:26,280 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020152844 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020152844-0000000000020153003
2018-04-25 15:33:26,807 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020153004 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153004-0000000000020153193
2018-04-25 15:35:27,146 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020153194 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153194-0000000000020153219
2018-04-25 15:35:27,517 INFO namenode.TransferFsImage (TransferFsImage.java:copyFileToStream(395)) - Sending fileName: /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153194-0000000000020153219, fileSize: 3505. Sent total: 3505 bytes. Size of last segment intended to send: -1 bytes.
2018-04-25 15:37:27,622 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020153220 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153220-0000000000020153428
2018-04-25 15:39:28,295 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020153429 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153429-0000000000020153786
2018-04-25 15:41:29,191 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020153787 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153787-0000000000020153867
2018-04-25 15:41:29,705 INFO namenode.TransferFsImage (TransferFsImage.java:copyFileToStream(395)) - Sending fileName: /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153787-0000000000020153867, fileSize: 12321. Sent total: 12321 bytes. Size of last segment intended to send: -1 bytes.
2018-04-25 15:43:29,817 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020153868 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153868-0000000000020153897
2018-04-25 15:44:23,499 INFO availability.MetricSinkWriteShardHostnameHashingStrategy (MetricSinkWriteShardHostnameHashingStrategy.java:findCollectorShard(42)) - Calculated collector shard nodo2.root.com.mx based on hostname: nodo1.root.com.mx
2018-04-25 15:45:30,133 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020153898 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020153898-0000000000020154350
2018-04-25 15:47:30,519 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020154351 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020154351-0000000000020154546
2018-04-25 15:49:30,981 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020154547 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020154547-0000000000020154555
2018-04-25 15:51:31,327 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020154556 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020154556-0000000000020154745
2018-04-25 15:53:31,885 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020154746 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020154746-0000000000020154752
2018-04-25 15:53:32,158 INFO namenode.TransferFsImage (TransferFsImage.java:copyFileToStream(395)) - Sending fileName: /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020154746-0000000000020154752, fileSize: 659. Sent total: 659 bytes. Size of last segment intended to send: -1 bytes.
2018-04-25 15:54:22,836 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file /hadoop/hdfs/journal/namenodeha/current/edits_inprogress_0000000000020154753 -> /hadoop/hdfs/journal/namenodeha/current/edits_0000000000020154753-0000000000020154755
2018-04-25 15:54:23,336 INFO common.Storage (JNStorage.java:purgeMatching(170)) - Purging no-longer needed file 19144244
2018-04-25 15:57:33,498 INFO timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:emitMetricsJson(199)) - Unable to connect to collector, http://nodo2.root.com.mx:6188/ws/v1/timeline/metrics
This exceptions will be ignored for next 100 times
2018-04-25 15:57:33,498 WARN timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://nodo2.root.com.mx:6188/ws/v1/timeline/metrics
2018-04-25 15:57:51,884 ERROR server.JournalNode (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2018-04-25 15:57:51,886 INFO server.JournalNode (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JournalNode at nodo1.root.com.mx/10.1.5.20
************************************************************/
==> /var/log/hadoop/hdfs/gc.log-201804021339 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(2003896k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T13:39:28.305-0500: 1.582: [GC (Allocation Failure) 2018-04-02T13:39:28.305-0500: 1.582: [ParNew: 104960K->11085K(118016K), 0.0118433 secs] 104960K->11085K(1035520K), 0.0119647 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-02T13:40:29.098-0500: 62.375: [GC (Allocation Failure) 2018-04-02T13:40:29.098-0500: 62.375: [ParNew: 116045K->13055K(118016K), 0.0712248 secs] 116045K->25487K(1035520K), 0.0713258 secs] [Times: user=0.25 sys=0.02, real=0.07 secs]
2018-04-02T13:40:29.170-0500: 62.447: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12431K(917504K)] 26052K(1035520K), 0.0072962 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-02T13:40:29.178-0500: 62.454: [CMS-concurrent-mark-start]
2018-04-02T13:40:29.186-0500: 62.463: [CMS-concurrent-mark: 0.009/0.009 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-02T13:40:29.186-0500: 62.463: [CMS-concurrent-preclean-start]
2018-04-02T13:40:29.189-0500: 62.465: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T13:40:29.189-0500: 62.466: [GC (CMS Final Remark) [YG occupancy: 14752 K (118016 K)]2018-04-02T13:40:29.189-0500: 62.466: [Rescan (parallel) , 0.0088687 secs]2018-04-02T13:40:29.198-0500: 62.475: [weak refs processing, 0.0000416 secs]2018-04-02T13:40:29.198-0500: 62.475: [class unloading, 0.0030495 secs]2018-04-02T13:40:29.201-0500: 62.478: [scrub symbol table, 0.0043091 secs]2018-04-02T13:40:29.205-0500: 62.482: [scrub string table, 0.0004368 secs][1 CMS-remark: 12431K(917504K)] 27183K(1035520K), 0.0173608 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-02T13:40:29.207-0500: 62.483: [CMS-concurrent-sweep-start]
2018-04-02T13:40:29.209-0500: 62.486: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T13:40:29.209-0500: 62.486: [CMS-concurrent-reset-start]
2018-04-02T13:40:29.217-0500: 62.493: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
2018-04-02T14:41:30.264-0500: 3723.540: [GC (Allocation Failure) 2018-04-02T14:41:30.264-0500: 3723.540: [ParNew: 118015K->3755K(118016K), 0.0496830 secs] 130442K->23162K(1035520K), 0.0497861 secs] [Times: user=0.25 sys=0.00, real=0.05 secs]
Heap
par new generation total 118016K, used 86417K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 78% used [0x00000000c0000000, 0x00000000c50b9758, 0x00000000c6680000)
from space 13056K, 28% used [0x00000000c7340000, 0x00000000c76eafd8, 0x00000000c8000000)
to space 13056K, 0% used [0x00000000c6680000, 0x00000000c6680000, 0x00000000c7340000)
concurrent mark-sweep generation total 917504K, used 19406K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25377K, capacity 25680K, committed 25932K, reserved 1073152K
class space used 2742K, capacity 2842K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804131017 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(7045392k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-13T10:17:46.400-0500: 1.877: [GC (GCLocker Initiated GC) 2018-04-13T10:17:46.400-0500: 1.877: [ParNew: 163840K->14106K(184320K), 0.0157274 secs] 163840K->14106K(1028096K), 0.0158445 secs] [Times: user=0.06 sys=0.01, real=0.01 secs]
2018-04-13T10:17:48.417-0500: 3.893: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 124129K(1028096K), 0.0158349 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-13T10:17:48.433-0500: 3.909: [CMS-concurrent-mark-start]
2018-04-13T10:17:48.444-0500: 3.921: [CMS-concurrent-mark: 0.012/0.012 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-13T10:17:48.444-0500: 3.921: [CMS-concurrent-preclean-start]
2018-04-13T10:17:48.448-0500: 3.925: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-13T10:17:48.448-0500: 3.925: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2018-04-13T10:17:53.541-0500: 9.018: [CMS-concurrent-abortable-preclean: 2.241/5.093 secs] [Times: user=2.28 sys=0.00, real=5.10 secs]
2018-04-13T10:17:53.541-0500: 9.018: [GC (CMS Final Remark) [YG occupancy: 129887 K (184320 K)]2018-04-13T10:17:53.541-0500: 9.018: [Rescan (parallel) , 0.0155009 secs]2018-04-13T10:17:53.557-0500: 9.034: [weak refs processing, 0.0000456 secs]2018-04-13T10:17:53.557-0500: 9.034: [class unloading, 0.0058197 secs]2018-04-13T10:17:53.563-0500: 9.040: [scrub symbol table, 0.0075392 secs]2018-04-13T10:17:53.571-0500: 9.047: [scrub string table, 0.0006773 secs][1 CMS-remark: 0K(843776K)] 129887K(1028096K), 0.0310080 secs] [Times: user=0.07 sys=0.01, real=0.03 secs]
2018-04-13T10:17:53.573-0500: 9.049: [CMS-concurrent-sweep-start]
2018-04-13T10:17:53.573-0500: 9.049: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-13T10:17:53.573-0500: 9.049: [CMS-concurrent-reset-start]
2018-04-13T10:17:53.577-0500: 9.054: [CMS-concurrent-reset: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-13T10:17:57.025-0500: 12.502: [GC (Allocation Failure) 2018-04-13T10:17:57.025-0500: 12.502: [ParNew: 177946K->11730K(184320K), 0.0399439 secs] 177946K->16170K(1028096K), 0.0401100 secs] [Times: user=0.15 sys=0.00, real=0.04 secs]
2018-04-13T10:31:46.862-0500: 842.339: [GC (Allocation Failure) 2018-04-13T10:31:46.862-0500: 842.339: [ParNew: 175570K->7249K(184320K), 0.0157313 secs] 180010K->16855K(1028096K), 0.0159045 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 7754K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 0% used [0x00000000c0000000, 0x00000000c007e188, 0x00000000ca000000)
from space 20480K, 35% used [0x00000000cb400000, 0x00000000cbb146a8, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 9605K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 31546K, capacity 31996K, committed 32344K, reserved 1077248K
class space used 3593K, capacity 3734K, committed 3756K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804051326 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(13841496k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-05T13:26:27.832-0500: 1.725: [GC (Allocation Failure) 2018-04-05T13:26:27.832-0500: 1.725: [ParNew: 104960K->11142K(118016K), 0.0140711 secs] 104960K->11142K(1035520K), 0.0141853 secs] [Times: user=0.05 sys=0.02, real=0.02 secs]
2018-04-05T13:27:29.409-0500: 63.303: [GC (Allocation Failure) 2018-04-05T13:27:29.409-0500: 63.303: [ParNew: 116102K->13056K(118016K), 0.0824334 secs] 116102K->25520K(1035520K), 0.0825338 secs] [Times: user=0.53 sys=0.02, real=0.08 secs]
2018-04-05T13:27:29.492-0500: 63.386: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12464K(917504K)] 26106K(1035520K), 0.0116647 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-05T13:27:29.504-0500: 63.397: [CMS-concurrent-mark-start]
2018-04-05T13:27:29.512-0500: 63.406: [CMS-concurrent-mark: 0.008/0.008 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-05T13:27:29.512-0500: 63.406: [CMS-concurrent-preclean-start]
2018-04-05T13:27:29.514-0500: 63.408: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-05T13:27:29.515-0500: 63.408: [GC (CMS Final Remark) [YG occupancy: 13641 K (118016 K)]2018-04-05T13:27:29.515-0500: 63.408: [Rescan (parallel) , 0.0073128 secs]2018-04-05T13:27:29.522-0500: 63.415: [weak refs processing, 0.0003946 secs]2018-04-05T13:27:29.522-0500: 63.416: [class unloading, 0.0032612 secs]2018-04-05T13:27:29.526-0500: 63.419: [scrub symbol table, 0.0044507 secs]2018-04-05T13:27:29.530-0500: 63.424: [scrub string table, 0.0004815 secs][1 CMS-remark: 12464K(917504K)] 26106K(1035520K), 0.0166610 secs] [Times: user=0.07 sys=0.00, real=0.01 secs]
2018-04-05T13:27:29.531-0500: 63.425: [CMS-concurrent-sweep-start]
2018-04-05T13:27:29.534-0500: 63.427: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2018-04-05T13:27:29.534-0500: 63.427: [CMS-concurrent-reset-start]
2018-04-05T13:27:29.538-0500: 63.431: [CMS-concurrent-reset: 0.004/0.004 secs] [Times: user=0.00 sys=0.01, real=0.00 secs]
2018-04-05T14:17:30.663-0500: 3064.556: [GC (Allocation Failure) 2018-04-05T14:17:30.663-0500: 3064.556: [ParNew: 118016K->3855K(118016K), 0.0507882 secs] 130474K->23242K(1035520K), 0.0509517 secs] [Times: user=0.38 sys=0.00, real=0.05 secs]
2018-04-05T16:30:31.178-0500: 11045.071: [GC (Allocation Failure) 2018-04-05T16:30:31.178-0500: 11045.071: [ParNew: 108815K->1792K(118016K), 0.0061106 secs] 128202K->21179K(1035520K), 0.0062290 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
Heap
par new generation total 118016K, used 21238K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 18% used [0x00000000c0000000, 0x00000000c12fd9b8, 0x00000000c6680000)
from space 13056K, 13% used [0x00000000c6680000, 0x00000000c68401f8, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 19387K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25429K, capacity 25696K, committed 26188K, reserved 1073152K
class space used 2738K, capacity 2842K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.1 <==
2018-04-16 14:29:11,422 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976251_3235453, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:11,601 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976252_3235454 src: /10.1.5.21:58678 dest: /10.1.5.20:50010
2018-04-16 14:29:11,603 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58678, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976252_3235454, duration: 952825
2018-04-16 14:29:11,603 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976252_3235454, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:11,766 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976253_3235455 src: /10.1.5.21:58682 dest: /10.1.5.20:50010
2018-04-16 14:29:11,768 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58682, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976253_3235455, duration: 960943
2018-04-16 14:29:11,768 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976253_3235455, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:11,933 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976254_3235456 src: /10.1.5.21:58686 dest: /10.1.5.20:50010
2018-04-16 14:29:11,935 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58686, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976254_3235456, duration: 964954
2018-04-16 14:29:11,935 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976254_3235456, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:12,313 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976255_3235457 src: /10.1.5.21:58692 dest: /10.1.5.20:50010
2018-04-16 14:29:12,315 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58692, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976255_3235457, duration: 1009167
2018-04-16 14:29:12,316 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976255_3235457, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:12,487 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976256_3235458 src: /10.1.5.21:58696 dest: /10.1.5.20:50010
2018-04-16 14:29:12,489 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58696, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976256_3235458, duration: 1153852
2018-04-16 14:29:12,489 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976256_3235458, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:12,652 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976257_3235459 src: /10.1.5.21:58700 dest: /10.1.5.20:50010
2018-04-16 14:29:12,654 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58700, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976257_3235459, duration: 1136417
2018-04-16 14:29:12,654 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976257_3235459, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:13,000 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976258_3235460 src: /10.1.5.21:58704 dest: /10.1.5.20:50010
2018-04-16 14:29:13,002 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58704, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976258_3235460, duration: 1070833
2018-04-16 14:29:13,002 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976258_3235460, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:13,180 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976259_3235461 src: /10.1.5.21:58710 dest: /10.1.5.20:50010
2018-04-16 14:29:13,182 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58710, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976259_3235461, duration: 1162866
2018-04-16 14:29:13,182 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976259_3235461, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:13,347 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976260_3235462 src: /10.1.5.21:58714 dest: /10.1.5.20:50010
2018-04-16 14:29:13,349 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58714, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976260_3235462, duration: 1193449
2018-04-16 14:29:13,349 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976260_3235462, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:13,512 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976261_3235463 src: /10.1.5.21:58718 dest: /10.1.5.20:50010
2018-04-16 14:29:13,514 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58718, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976261_3235463, duration: 1181711
2018-04-16 14:29:13,514 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976261_3235463, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:13,884 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976262_3235464 src: /10.1.5.21:58724 dest: /10.1.5.20:50010
2018-04-16 14:29:13,886 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58724, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976262_3235464, duration: 1204573
2018-04-16 14:29:13,886 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976262_3235464, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:14,142 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976263_3235465 src: /10.1.5.21:58728 dest: /10.1.5.20:50010
2018-04-16 14:29:14,144 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58728, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976263_3235465, duration: 1141377
2018-04-16 14:29:14,144 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976263_3235465, type=LAST_IN_PIPELINE terminating
2018-04-16 14:29:14,309 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076976264_3235466 src: /10.1.5.21:58734 dest: /10.1.5.20:50010
2018-04-16 14:29:14,311 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:58734, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076976264_3235466, duration: 1212962
2018-04-16 14:29:14,311 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076976264_3235466, type=LAST_IN_PIPELINE terminating
==> /var/log/hadoop/hdfs/gc.log-201804251827 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(14468000k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T18:27:55.459-0500: 1.667: [GC (Allocation Failure) 2018-04-25T18:27:55.460-0500: 1.667: [ParNew: 163840K->12701K(184320K), 0.0136721 secs] 163840K->12701K(1028096K), 0.0138042 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
Heap
par new generation total 184320K, used 148559K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 82% used [0x00000000c0000000, 0x00000000c84ac8f0, 0x00000000ca000000)
from space 20480K, 62% used [0x00000000cb400000, 0x00000000cc067428, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25010K, capacity 25312K, committed 25612K, reserved 1071104K
class space used 2980K, capacity 3103K, committed 3168K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-nodo1.root.com.mx.out.5 <==
ulimit -a for user hdfs
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804251906 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(13974056k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T19:06:03.695-0500: 1.695: [GC (Allocation Failure) 2018-04-25T19:06:03.695-0500: 1.695: [ParNew: 163840K->12704K(184320K), 0.0139020 secs] 163840K->12704K(1028096K), 0.0140451 secs] [Times: user=0.03 sys=0.01, real=0.02 secs]
Heap
par new generation total 184320K, used 149523K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 83% used [0x00000000c0000000, 0x00000000c859cd70, 0x00000000ca000000)
from space 20480K, 62% used [0x00000000cb400000, 0x00000000cc0681f0, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25012K, capacity 25300K, committed 25536K, reserved 1071104K
class space used 2983K, capacity 3103K, committed 3144K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804101001 <==
2018-04-12T19:33:03.474-0500: 207075.873: [GC (Allocation Failure) 2018-04-12T19:33:03.474-0500: 207075.873: [ParNew: 165271K->1021K(184320K), 0.0074183 secs] 179663K->15414K(1028096K), 0.0076169 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-12T20:20:29.364-0500: 209921.764: [GC (Allocation Failure) 2018-04-12T20:20:29.364-0500: 209921.764: [ParNew: 164861K->1250K(184320K), 0.0079623 secs] 179254K->15644K(1028096K), 0.0081564 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-12T21:07:54.294-0500: 212766.694: [GC (Allocation Failure) 2018-04-12T21:07:54.295-0500: 212766.695: [ParNew: 165090K->1235K(184320K), 0.0081309 secs] 179484K->15629K(1028096K), 0.0090538 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-12T21:55:11.599-0500: 215603.998: [GC (Allocation Failure) 2018-04-12T21:55:11.599-0500: 215603.999: [ParNew: 165075K->1247K(184320K), 0.0079936 secs] 179469K->15641K(1028096K), 0.0081713 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-12T22:42:52.071-0500: 218464.470: [GC (Allocation Failure) 2018-04-12T22:42:52.071-0500: 218464.470: [ParNew: 165087K->1174K(184320K), 0.0078086 secs] 179481K->15571K(1028096K), 0.0079499 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-12T23:30:20.165-0500: 221312.564: [GC (Allocation Failure) 2018-04-12T23:30:20.165-0500: 221312.564: [ParNew: 165014K->1144K(184320K), 0.0078661 secs] 179411K->15607K(1028096K), 0.0080583 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T00:17:59.491-0500: 224171.890: [GC (Allocation Failure) 2018-04-13T00:17:59.491-0500: 224171.890: [ParNew: 164984K->1279K(184320K), 0.0078803 secs] 179447K->15778K(1028096K), 0.0080664 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T01:04:51.444-0500: 226983.843: [GC (Allocation Failure) 2018-04-13T01:04:51.444-0500: 226983.843: [ParNew: 165119K->1359K(184320K), 0.0080413 secs] 179618K->15859K(1028096K), 0.0082123 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-13T01:43:31.034-0500: 229303.434: [GC (Allocation Failure) 2018-04-13T01:43:31.034-0500: 229303.434: [ParNew: 165199K->1306K(184320K), 0.0086968 secs] 179699K->15807K(1028096K), 0.0089063 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T02:30:29.491-0500: 232121.890: [GC (Allocation Failure) 2018-04-13T02:30:29.491-0500: 232121.890: [ParNew: 165146K->1420K(184320K), 0.0065569 secs] 179647K->15920K(1028096K), 0.0067562 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-13T03:17:22.553-0500: 234934.953: [GC (Allocation Failure) 2018-04-13T03:17:22.553-0500: 234934.953: [ParNew: 165260K->1244K(184320K), 0.0092113 secs] 179760K->15907K(1028096K), 0.0093865 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T04:05:12.512-0500: 237804.911: [GC (Allocation Failure) 2018-04-13T04:05:12.512-0500: 237804.912: [ParNew: 165084K->1206K(184320K), 0.0087775 secs] 179747K->15868K(1028096K), 0.0089588 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T04:52:47.781-0500: 240660.180: [GC (Allocation Failure) 2018-04-13T04:52:47.781-0500: 240660.180: [ParNew: 165046K->1115K(184320K), 0.0089940 secs] 179708K->15927K(1028096K), 0.0091724 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T05:39:50.974-0500: 243483.373: [GC (Allocation Failure) 2018-04-13T05:39:50.974-0500: 243483.373: [ParNew: 164955K->1018K(184320K), 0.0079700 secs] 179767K->15831K(1028096K), 0.0081496 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T06:27:22.565-0500: 246334.964: [GC (Allocation Failure) 2018-04-13T06:27:22.565-0500: 246334.964: [ParNew: 164858K->912K(184320K), 0.0082272 secs] 179671K->15725K(1028096K), 0.0083845 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T07:14:29.489-0500: 249161.889: [GC (Allocation Failure) 2018-04-13T07:14:29.489-0500: 249161.889: [ParNew: 164752K->920K(184320K), 0.0078267 secs] 179565K->15799K(1028096K), 0.0079928 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-13T07:52:49.496-0500: 251461.895: [GC (Allocation Failure) 2018-04-13T07:52:49.496-0500: 251461.895: [ParNew: 164760K->858K(184320K), 0.0086032 secs] 179639K->15738K(1028096K), 0.0088064 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T08:40:30.398-0500: 254322.797: [GC (Allocation Failure) 2018-04-13T08:40:30.398-0500: 254322.798: [ParNew: 164698K->861K(184320K), 0.0087753 secs] 179578K->15741K(1028096K), 0.0089590 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-13T09:27:31.352-0500: 257143.752: [GC (Allocation Failure) 2018-04-13T09:27:31.352-0500: 257143.752: [ParNew: 164701K->850K(184320K), 0.0039055 secs] 179581K->15796K(1028096K), 0.0040225 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-13T10:13:24.165-0500: 259896.565: [GC (CMS Initial Mark) [1 CMS-initial-mark: 14945K(843776K)] 178128K(1028096K), 0.0689377 secs] [Times: user=0.27 sys=0.00, real=0.07 secs]
2018-04-13T10:13:24.234-0500: 259896.634: [CMS-concurrent-mark-start]
2018-04-13T10:13:24.269-0500: 259896.669: [CMS-concurrent-mark: 0.035/0.035 secs] [Times: user=0.03 sys=0.00, real=0.04 secs]
2018-04-13T10:13:24.269-0500: 259896.669: [CMS-concurrent-preclean-start]
2018-04-13T10:13:24.273-0500: 259896.673: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-13T10:13:24.273-0500: 259896.673: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2018-04-13T10:13:29.286-0500: 259901.686: [CMS-concurrent-abortable-preclean: 0.407/5.013 secs] [Times: user=0.42 sys=0.00, real=5.01 secs]
2018-04-13T10:13:29.287-0500: 259901.686: [GC (CMS Final Remark) [YG occupancy: 163312 K (184320 K)]2018-04-13T10:13:29.287-0500: 259901.686: [Rescan (parallel) , 0.0653282 secs]2018-04-13T10:13:29.352-0500: 259901.752: [weak refs processing, 0.0001948 secs]2018-04-13T10:13:29.353-0500: 259901.752: [class unloading, 0.0272174 secs]2018-04-13T10:13:29.380-0500: 259901.779: [scrub symbol table, 0.0064158 secs]2018-04-13T10:13:29.386-0500: 259901.786: [scrub string table, 0.0009959 secs][1 CMS-remark: 14945K(843776K)] 178257K(1028096K), 0.1023427 secs] [Times: user=0.30 sys=0.00, real=0.11 secs]
2018-04-13T10:13:29.390-0500: 259901.789: [CMS-concurrent-sweep-start]
2018-04-13T10:13:29.399-0500: 259901.799: [CMS-concurrent-sweep: 0.010/0.010 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-13T10:13:29.399-0500: 259901.799: [CMS-concurrent-reset-start]
2018-04-13T10:13:29.402-0500: 259901.802: [CMS-concurrent-reset: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-13T10:13:43.238-0500: 259915.637: [GC (Allocation Failure) 2018-04-13T10:13:43.238-0500: 259915.637: [ParNew: 164690K->539K(184320K), 0.0069295 secs] 176638K->12487K(1028096K), 0.0070838 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 13894K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 8% used [0x00000000c0000000, 0x00000000c0d0ad30, 0x00000000ca000000)
from space 20480K, 2% used [0x00000000ca000000, 0x00000000ca086d88, 0x00000000cb400000)
to space 20480K, 0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
concurrent mark-sweep generation total 843776K, used 11948K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 43505K, capacity 44262K, committed 44604K, reserved 1089536K
class space used 4543K, capacity 4747K, committed 4780K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.8 <==
2018-04-05 00:00:08,092 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580109_1839295 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580109 for deletion
2018-04-05 00:00:08,092 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580105_1839291 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580105
2018-04-05 00:00:08,092 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580110_1839296 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580110 for deletion
2018-04-05 00:00:08,092 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580106_1839292 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580106
2018-04-05 00:00:08,092 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580111_1839297 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580111 for deletion
2018-04-05 00:00:08,092 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580107_1839293 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580107
2018-04-05 00:00:08,092 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580112_1839298 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580112 for deletion
2018-04-05 00:00:08,093 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580108_1839294 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580108
2018-04-05 00:00:08,093 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580113_1839299 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580113 for deletion
2018-04-05 00:00:08,093 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580109_1839295 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580109
2018-04-05 00:00:08,093 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580114_1839300 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580114 for deletion
2018-04-05 00:00:08,093 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580110_1839296 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580110
2018-04-05 00:00:08,093 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580115_1839301 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580115 for deletion
2018-04-05 00:00:08,093 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580111_1839297 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580111
2018-04-05 00:00:08,093 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580116_1839302 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580116 for deletion
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580112_1839298 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580112
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580117_1839303 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580117 for deletion
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580113_1839299 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580113
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580118_1839304 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580118 for deletion
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580114_1839300 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580114
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580119_1839305 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580119 for deletion
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580115_1839301 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580115
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580120_1839306 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580120 for deletion
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580116_1839302 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580116
2018-04-05 00:00:08,094 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580121_1839307 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580121 for deletion
2018-04-05 00:00:08,095 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580117_1839303 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580117
2018-04-05 00:00:08,095 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580122_1839308 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580122 for deletion
2018-04-05 00:00:08,095 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580118_1839304 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580118
2018-04-05 00:00:08,095 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580123_1839309 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580123 for deletion
2018-04-05 00:00:08,095 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580119_1839305 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580119
2018-04-05 00:00:08,095 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580124_1839310 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580124 for deletion
2018-04-05 00:00:08,095 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580120_1839306 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580120
2018-04-05 00:00:08,095 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580125_1839311 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580125 for deletion
2018-04-05 00:00:08,096 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580121_1839307 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580121
2018-04-05 00:00:08,096 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580126_1839312 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580126 for deletion
2018-04-05 00:00:08,096 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580122_1839308 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580122
2018-04-05 00:00:08,096 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580127_1839313 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580127 for deletion
2018-04-05 00:00:08,096 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580123_1839309 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580123
2018-04-05 00:00:08,096 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075580128_1839314 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580128 for deletion
2018-04-05 00:00:08,096 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075580124_1839310 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir28/subdir12/blk_1075580124
==> /var/log/hadoop/hdfs/gc.log-201804131021 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(9587128k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-13T10:21:38.138-0500: 1.397: [GC (Allocation Failure) 2018-04-13T10:21:38.138-0500: 1.397: [ParNew: 104960K->11081K(118016K), 0.0133064 secs] 104960K->11081K(1035520K), 0.0134299 secs] [Times: user=0.04 sys=0.02, real=0.02 secs]
2018-04-13T10:22:39.336-0500: 62.596: [GC (Allocation Failure) 2018-04-13T10:22:39.336-0500: 62.596: [ParNew: 116041K->13056K(118016K), 0.0807413 secs] 116041K->26023K(1035520K), 0.0808838 secs] [Times: user=0.53 sys=0.02, real=0.08 secs]
2018-04-13T10:22:39.417-0500: 62.677: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12967K(917504K)] 26582K(1035520K), 0.0086311 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
2018-04-13T10:22:39.426-0500: 62.686: [CMS-concurrent-mark-start]
2018-04-13T10:22:39.437-0500: 62.697: [CMS-concurrent-mark: 0.011/0.011 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]
2018-04-13T10:22:39.437-0500: 62.697: [CMS-concurrent-preclean-start]
2018-04-13T10:22:39.442-0500: 62.702: [CMS-concurrent-preclean: 0.005/0.005 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T10:22:39.442-0500: 62.702: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2018-04-13T10:22:44.559-0500: 67.819: [CMS-concurrent-abortable-preclean: 1.508/5.117 secs] [Times: user=3.00 sys=0.05, real=5.12 secs]
2018-04-13T10:22:44.560-0500: 67.819: [GC (CMS Final Remark) [YG occupancy: 63605 K (118016 K)]2018-04-13T10:22:44.560-0500: 67.819: [Rescan (parallel) , 0.0185399 secs]2018-04-13T10:22:44.578-0500: 67.838: [weak refs processing, 0.0000761 secs]2018-04-13T10:22:44.578-0500: 67.838: [class unloading, 0.0066623 secs]2018-04-13T10:22:44.585-0500: 67.845: [scrub symbol table, 0.0073800 secs]2018-04-13T10:22:44.592-0500: 67.852: [scrub string table, 0.0009128 secs][1 CMS-remark: 12967K(917504K)] 76572K(1035520K), 0.0352415 secs] [Times: user=0.17 sys=0.00, real=0.03 secs]
2018-04-13T10:22:44.595-0500: 67.855: [CMS-concurrent-sweep-start]
2018-04-13T10:22:44.598-0500: 67.857: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-13T10:22:44.598-0500: 67.857: [CMS-concurrent-reset-start]
2018-04-13T10:22:44.601-0500: 67.861: [CMS-concurrent-reset: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
Heap
par new generation total 118016K, used 77548K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 61% used [0x00000000c0000000, 0x00000000c3efb258, 0x00000000c6680000)
from space 13056K, 100% used [0x00000000c6680000, 0x00000000c7340000, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 12962K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 24978K, capacity 25250K, committed 25700K, reserved 1073152K
class space used 2738K, capacity 2842K, committed 2916K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.6 <==
2018-04-14 08:00:52,169 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977893_2237094 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977893 for deletion
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977894_2237095 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977894 for deletion
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977873_2237074 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977873
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977895_2237096 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977895 for deletion
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977896_2237097 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977896 for deletion
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977874_2237075 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977874
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977897_2237098 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977897 for deletion
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977898_2237099 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977898 for deletion
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977875_2237076 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977875
2018-04-14 08:00:52,170 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977899_2237100 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977899 for deletion
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977900_2237101 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977900 for deletion
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977876_2237077 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977876
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977901_2237102 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977901 for deletion
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977902_2237103 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977902 for deletion
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977877_2237078 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977877
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977903_2237104 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977903 for deletion
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977904_2237105 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977904 for deletion
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977878_2237079 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977878
2018-04-14 08:00:52,171 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075977905_2237106 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977905 for deletion
2018-04-14 08:00:52,172 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977879_2237080 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977879
2018-04-14 08:00:52,172 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977880_2237081 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977880
2018-04-14 08:00:52,172 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977881_2237082 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977881
2018-04-14 08:00:52,172 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977882_2237083 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977882
2018-04-14 08:00:52,172 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977883_2237084 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977883
2018-04-14 08:00:52,173 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977884_2237085 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977884
2018-04-14 08:00:52,173 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977885_2237086 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977885
2018-04-14 08:00:52,173 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977886_2237087 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977886
2018-04-14 08:00:52,173 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977887_2237088 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977887
2018-04-14 08:00:52,174 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977888_2237089 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977888
2018-04-14 08:00:52,174 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977889_2237090 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977889
2018-04-14 08:00:52,174 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977890_2237091 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977890
2018-04-14 08:00:52,174 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977891_2237092 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977891
2018-04-14 08:00:52,174 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977892_2237093 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977892
2018-04-14 08:00:52,175 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977893_2237094 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977893
2018-04-14 08:00:52,175 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977894_2237095 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977894
2018-04-14 08:00:52,175 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977895_2237096 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977895
2018-04-14 08:00:52,175 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977896_2237097 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977896
2018-04-14 08:00:52,175 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977897_2237098 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977897
2018-04-14 08:00:52,176 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977898_2237099 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977898
2018-04-14 08:00:52,176 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075977899_2237100 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir34/subdir30/blk_1075977899
==> /var/log/hadoop/hdfs/gc.log-201804171011 <==
2018-04-25T13:17:04.627-0500: 702340.696: [GC (Allocation Failure) 2018-04-25T13:17:04.627-0500: 702340.696: [ParNew: 166078K->2499K(184320K), 0.0074114 secs] 184225K->20647K(1028096K), 0.0075982 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T13:25:44.531-0500: 702860.600: [GC (Allocation Failure) 2018-04-25T13:25:44.531-0500: 702860.600: [ParNew: 166339K->2288K(184320K), 0.0069049 secs] 184487K->20435K(1028096K), 0.0070879 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T13:26:35.943-0500: 702912.012: [GC (Allocation Failure) 2018-04-25T13:26:35.943-0500: 702912.012: [ParNew: 166128K->2341K(184320K), 0.0059002 secs] 184275K->20489K(1028096K), 0.0060952 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T13:34:21.243-0500: 703377.312: [GC (Allocation Failure) 2018-04-25T13:34:21.243-0500: 703377.312: [ParNew: 166181K->3588K(184320K), 0.0105550 secs] 184329K->21745K(1028096K), 0.0107414 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T13:34:31.758-0500: 703387.827: [GC (Allocation Failure) 2018-04-25T13:34:31.759-0500: 703387.828: [ParNew: 167428K->3868K(184320K), 0.0115620 secs] 185585K->22029K(1028096K), 0.0117452 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T13:34:51.962-0500: 703408.031: [GC (Allocation Failure) 2018-04-25T13:34:51.962-0500: 703408.031: [ParNew: 167708K->3174K(184320K), 0.0065462 secs] 185869K->21401K(1028096K), 0.0066508 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T13:35:22.649-0500: 703438.718: [GC (Allocation Failure) 2018-04-25T13:35:22.649-0500: 703438.718: [ParNew: 167014K->3260K(184320K), 0.0059998 secs] 185241K->21560K(1028096K), 0.0061469 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T13:56:56.897-0500: 704732.966: [GC (Allocation Failure) 2018-04-25T13:56:56.897-0500: 704732.966: [ParNew: 167100K->1913K(184320K), 0.0085745 secs] 185400K->20278K(1028096K), 0.0087701 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T14:35:57.929-0500: 707073.998: [GC (Allocation Failure) 2018-04-25T14:35:57.929-0500: 707073.998: [ParNew: 165753K->1660K(184320K), 0.0050041 secs] 184118K->20026K(1028096K), 0.0051745 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-25T15:01:08.136-0500: 708584.205: [GC (Allocation Failure) 2018-04-25T15:01:08.136-0500: 708584.205: [ParNew: 165500K->1309K(184320K), 0.0058271 secs] 183866K->19742K(1028096K), 0.0060021 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-25T15:15:22.874-0500: 709438.943: [GC (Allocation Failure) 2018-04-25T15:15:22.874-0500: 709438.943: [ParNew: 165149K->1784K(184320K), 0.0032903 secs] 183582K->20218K(1028096K), 0.0033925 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-25T15:16:10.891-0500: 709486.960: [GC (Allocation Failure) 2018-04-25T15:16:10.891-0500: 709486.960: [ParNew: 165624K->4099K(184320K), 0.0066502 secs] 184058K->22534K(1028096K), 0.0068010 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T15:28:41.507-0500: 710237.576: [GC (Allocation Failure) 2018-04-25T15:28:41.507-0500: 710237.576: [ParNew: 167939K->3147K(184320K), 0.0053887 secs] 186374K->21582K(1028096K), 0.0054990 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-25T15:29:36.023-0500: 710292.092: [GC (GCLocker Initiated GC) 2018-04-25T15:29:36.023-0500: 710292.092: [ParNew: 166987K->3606K(184320K), 0.0101088 secs] 185422K->22041K(1028096K), 0.0102807 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T15:29:43.271-0500: 710299.341: [GC (Allocation Failure) 2018-04-25T15:29:43.272-0500: 710299.341: [ParNew: 167446K->3788K(184320K), 0.0131291 secs] 185881K->22224K(1028096K), 0.0133256 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2018-04-25T15:29:49.419-0500: 710305.488: [GC (Allocation Failure) 2018-04-25T15:29:49.419-0500: 710305.488: [ParNew: 167628K->4130K(184320K), 0.0136352 secs] 186064K->22565K(1028096K), 0.0137942 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T15:30:04.539-0500: 710320.608: [GC (Allocation Failure) 2018-04-25T15:30:04.539-0500: 710320.608: [ParNew: 167970K->3292K(184320K), 0.0128828 secs] 186405K->21764K(1028096K), 0.0130611 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T15:30:18.497-0500: 710334.566: [GC (Allocation Failure) 2018-04-25T15:30:18.497-0500: 710334.566: [ParNew: 167132K->3198K(184320K), 0.0121706 secs] 185604K->21722K(1028096K), 0.0123497 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T15:30:36.529-0500: 710352.598: [GC (Allocation Failure) 2018-04-25T15:30:36.529-0500: 710352.598: [ParNew: 167038K->2730K(184320K), 0.0108502 secs] 185562K->21606K(1028096K), 0.0110521 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T15:30:47.568-0500: 710363.638: [GC (Allocation Failure) 2018-04-25T15:30:47.569-0500: 710363.638: [ParNew: 166570K->2682K(184320K), 0.0130469 secs] 185446K->21926K(1028096K), 0.0132224 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-25T15:37:08.488-0500: 710744.557: [GC (Allocation Failure) 2018-04-25T15:37:08.488-0500: 710744.557: [ParNew: 166522K->2717K(184320K), 0.0097087 secs] 185766K->21961K(1028096K), 0.0098883 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T15:38:49.399-0500: 710845.468: [GC (Allocation Failure) 2018-04-25T15:38:49.399-0500: 710845.468: [ParNew: 166557K->3295K(184320K), 0.0073261 secs] 185801K->22542K(1028096K), 0.0074832 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T15:39:04.062-0500: 710860.131: [GC (Allocation Failure) 2018-04-25T15:39:04.062-0500: 710860.132: [ParNew: 167135K->2913K(184320K), 0.0093788 secs] 186382K->22161K(1028096K), 0.0095559 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T15:39:14.328-0500: 710870.397: [GC (Allocation Failure) 2018-04-25T15:39:14.328-0500: 710870.397: [ParNew: 166753K->3209K(184320K), 0.0100763 secs] 186001K->22458K(1028096K), 0.0102264 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T15:39:25.816-0500: 710881.885: [GC (Allocation Failure) 2018-04-25T15:39:25.816-0500: 710881.885: [ParNew: 167047K->3390K(184320K), 0.0076761 secs] 186297K->22640K(1028096K), 0.0078089 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-25T15:39:44.827-0500: 710900.896: [GC (Allocation Failure) 2018-04-25T15:39:44.827-0500: 710900.896: [ParNew: 167230K->2846K(184320K), 0.0107086 secs] 186480K->22096K(1028096K), 0.0109011 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-25T15:45:50.355-0500: 711266.425: [GC (Allocation Failure) 2018-04-25T15:45:50.356-0500: 711266.425: [ParNew: 166686K->3873K(184320K), 0.0053318 secs] 185936K->23164K(1028096K), 0.0054406 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-25T15:45:59.907-0500: 711275.976: [GC (Allocation Failure) 2018-04-25T15:45:59.907-0500: 711275.976: [ParNew: 167713K->3490K(184320K), 0.0094590 secs] 187004K->22850K(1028096K), 0.0096378 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T15:46:14.527-0500: 711290.596: [GC (Allocation Failure) 2018-04-25T15:46:14.527-0500: 711290.596: [ParNew: 167330K->3304K(184320K), 0.0094337 secs] 186690K->22729K(1028096K), 0.0095804 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T15:46:29.765-0500: 711305.834: [GC (Allocation Failure) 2018-04-25T15:46:29.766-0500: 711305.835: [ParNew: 167144K->3185K(184320K), 0.0067508 secs] 186569K->22613K(1028096K), 0.0068948 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-25T15:46:46.926-0500: 711322.995: [GC (Allocation Failure) 2018-04-25T15:46:46.926-0500: 711322.995: [ParNew: 167025K->3260K(184320K), 0.0084479 secs] 186453K->22688K(1028096K), 0.0086347 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-25T15:51:46.896-0500: 711622.965: [GC (Allocation Failure) 2018-04-25T15:51:46.896-0500: 711622.965: [ParNew: 167100K->3026K(184320K), 0.0092806 secs] 186528K->22456K(1028096K), 0.0094628 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 31301K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 17% used [0x00000000c0000000, 0x00000000c1b9cd18, 0x00000000ca000000)
from space 20480K, 14% used [0x00000000ca000000, 0x00000000ca2f4ab0, 0x00000000cb400000)
to space 20480K, 0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
concurrent mark-sweep generation total 843776K, used 19429K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 45729K, capacity 46514K, committed 46864K, reserved 1091584K
class space used 4693K, capacity 4897K, committed 4940K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804251902 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(14231896k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T19:02:45.767-0500: 1.381: [GC (Allocation Failure) 2018-04-25T19:02:45.767-0500: 1.381: [ParNew: 104960K->11168K(118016K), 0.0151110 secs] 104960K->11168K(1035520K), 0.0152302 secs] [Times: user=0.05 sys=0.02, real=0.02 secs]
2018-04-25T19:03:46.760-0500: 62.374: [GC (Allocation Failure) 2018-04-25T19:03:46.760-0500: 62.374: [ParNew: 116128K->13056K(118016K), 0.0844694 secs] 116128K->25329K(1035520K), 0.0845801 secs] [Times: user=0.42 sys=0.02, real=0.08 secs]
2018-04-25T19:03:46.846-0500: 62.460: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12273K(917504K)] 25909K(1035520K), 0.0080058 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T19:03:46.854-0500: 62.468: [CMS-concurrent-mark-start]
2018-04-25T19:03:46.865-0500: 62.479: [CMS-concurrent-mark: 0.011/0.011 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-25T19:03:46.865-0500: 62.479: [CMS-concurrent-preclean-start]
2018-04-25T19:03:46.868-0500: 62.482: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-25T19:03:46.868-0500: 62.482: [GC (CMS Final Remark) [YG occupancy: 13635 K (118016 K)]2018-04-25T19:03:46.868-0500: 62.482: [Rescan (parallel) , 0.0071326 secs]2018-04-25T19:03:46.875-0500: 62.489: [weak refs processing, 0.0003474 secs]2018-04-25T19:03:46.875-0500: 62.489: [class unloading, 0.0029197 secs]2018-04-25T19:03:46.878-0500: 62.492: [scrub symbol table, 0.0039757 secs]2018-04-25T19:03:46.882-0500: 62.496: [scrub string table, 0.0004286 secs][1 CMS-remark: 12273K(917504K)] 25909K(1035520K), 0.0154183 secs] [Times: user=0.05 sys=0.01, real=0.01 secs]
2018-04-25T19:03:46.883-0500: 62.497: [CMS-concurrent-sweep-start]
2018-04-25T19:03:46.886-0500: 62.500: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-25T19:03:46.886-0500: 62.500: [CMS-concurrent-reset-start]
2018-04-25T19:03:46.891-0500: 62.505: [CMS-concurrent-reset: 0.005/0.005 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
==> /var/log/hadoop/hdfs/gc.log-201804051324 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(14616344k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-05T13:24:42.379-0500: 8.623: [GC (GCLocker Initiated GC) 2018-04-05T13:24:42.380-0500: 8.624: [ParNew: 163840K->14118K(184320K), 0.0222818 secs] 163840K->14118K(1028096K), 0.0224508 secs] [Times: user=0.08 sys=0.02, real=0.02 secs]
2018-04-05T13:24:44.402-0500: 10.646: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 118469K(1028096K), 0.0128875 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-05T13:24:44.415-0500: 10.659: [CMS-concurrent-mark-start]
2018-04-05T13:24:44.421-0500: 10.665: [CMS-concurrent-mark: 0.005/0.005 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-05T13:24:44.421-0500: 10.665: [CMS-concurrent-preclean-start]
2018-04-05T13:24:44.423-0500: 10.667: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2018-04-05T13:24:44.423-0500: 10.667: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2018-04-05T13:24:49.607-0500: 15.851: [CMS-concurrent-abortable-preclean: 2.664/5.184 secs] [Times: user=4.16 sys=0.72, real=5.18 secs]
2018-04-05T13:24:49.609-0500: 15.853: [GC (CMS Final Remark) [YG occupancy: 170583 K (184320 K)]2018-04-05T13:24:49.609-0500: 15.853: [Rescan (parallel) , 0.0234172 secs]2018-04-05T13:24:49.633-0500: 15.877: [weak refs processing, 0.0000509 secs]2018-04-05T13:24:49.633-0500: 15.877: [class unloading, 0.0063397 secs]2018-04-05T13:24:49.639-0500: 15.883: [scrub symbol table, 0.0086880 secs]2018-04-05T13:24:49.648-0500: 15.892: [scrub string table, 0.0008081 secs][1 CMS-remark: 0K(843776K)] 170583K(1028096K), 0.0405573 secs] [Times: user=0.11 sys=0.00, real=0.04 secs]
2018-04-05T13:24:49.650-0500: 15.894: [CMS-concurrent-sweep-start]
2018-04-05T13:24:49.650-0500: 15.894: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-05T13:24:49.650-0500: 15.894: [CMS-concurrent-reset-start]
2018-04-05T13:24:49.655-0500: 15.899: [CMS-concurrent-reset: 0.005/0.005 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-05T13:24:51.968-0500: 18.212: [GC (Allocation Failure) 2018-04-05T13:24:51.968-0500: 18.212: [ParNew: 177958K->13563K(184320K), 0.0419369 secs] 177958K->18000K(1028096K), 0.0420895 secs] [Times: user=0.13 sys=0.00, real=0.04 secs]
2018-04-05T13:32:06.991-0500: 453.235: [GC (Allocation Failure) 2018-04-05T13:32:06.991-0500: 453.235: [ParNew: 177380K->8050K(184320K), 0.0236017 secs] 181816K->17829K(1028096K), 0.0237816 secs] [Times: user=0.08 sys=0.01, real=0.03 secs]
2018-04-05T14:11:20.439-0500: 2806.683: [GC (Allocation Failure) 2018-04-05T14:11:20.439-0500: 2806.683: [ParNew: 171890K->2369K(184320K), 0.0104533 secs] 181669K->12148K(1028096K), 0.0106359 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-05T14:46:36.954-0500: 4923.198: [GC (Allocation Failure) 2018-04-05T14:46:36.955-0500: 4923.199: [ParNew: 166209K->2500K(184320K), 0.0095572 secs] 175988K->12279K(1028096K), 0.0097566 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-05T15:29:35.332-0500: 7501.576: [GC (Allocation Failure) 2018-04-05T15:29:35.332-0500: 7501.576: [ParNew: 166340K->2513K(184320K), 0.0095378 secs] 176119K->12292K(1028096K), 0.0097356 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T16:17:11.958-0500: 10358.202: [GC (Allocation Failure) 2018-04-05T16:17:11.958-0500: 10358.202: [ParNew: 166339K->2376K(184320K), 0.0078021 secs] 176118K->12154K(1028096K), 0.0080201 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T16:53:32.110-0500: 12538.354: [GC (Allocation Failure) 2018-04-05T16:53:32.110-0500: 12538.354: [ParNew: 166216K->3945K(184320K), 0.0085743 secs] 175994K->13724K(1028096K), 0.0087372 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 13681K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 5% used [0x00000000c0000000, 0x00000000c09820e8, 0x00000000ca000000)
from space 20480K, 19% used [0x00000000ca000000, 0x00000000ca3da6d0, 0x00000000cb400000)
to space 20480K, 0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
concurrent mark-sweep generation total 843776K, used 9778K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 36292K, capacity 36722K, committed 37024K, reserved 1083392K
class space used 3902K, capacity 4047K, committed 4064K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804131043 <==
2018-04-16T17:35:08.772-0500: 283870.950: [GC (Allocation Failure) 2018-04-16T17:35:08.773-0500: 283870.950: [ParNew: 166731K->2565K(184320K), 0.0169941 secs] 256069K->91988K(1028096K), 0.0171755 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2018-04-16T17:38:39.672-0500: 284081.849: [GC (Allocation Failure) 2018-04-16T17:38:39.672-0500: 284081.849: [ParNew: 166405K->2763K(184320K), 0.0208494 secs] 255828K->92271K(1028096K), 0.0210579 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-16T17:42:11.275-0500: 284293.453: [GC (Allocation Failure) 2018-04-16T17:42:11.276-0500: 284293.453: [ParNew: 166603K->2503K(184320K), 0.0215926 secs] 256111K->92219K(1028096K), 0.0217797 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T17:45:43.792-0500: 284505.969: [GC (Allocation Failure) 2018-04-16T17:45:43.792-0500: 284505.970: [ParNew: 166343K->2541K(184320K), 0.0217575 secs] 256059K->92342K(1028096K), 0.0219587 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-16T17:49:19.868-0500: 284722.045: [GC (Allocation Failure) 2018-04-16T17:49:19.868-0500: 284722.045: [ParNew: 166381K->2860K(184320K), 0.0164293 secs] 256182K->92781K(1028096K), 0.0165903 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2018-04-16T17:52:51.956-0500: 284934.133: [GC (Allocation Failure) 2018-04-16T17:52:51.956-0500: 284934.133: [ParNew: 166700K->2544K(184320K), 0.0239107 secs] 256621K->92550K(1028096K), 0.0241168 secs] [Times: user=0.07 sys=0.00, real=0.03 secs]
2018-04-16T17:56:19.219-0500: 285141.396: [GC (Allocation Failure) 2018-04-16T17:56:19.219-0500: 285141.396: [ParNew: 166384K->2733K(184320K), 0.0136063 secs] 256390K->92826K(1028096K), 0.0137347 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-16T17:59:46.202-0500: 285348.380: [GC (Allocation Failure) 2018-04-16T17:59:46.202-0500: 285348.380: [ParNew: 166573K->2366K(184320K), 0.0216964 secs] 256666K->92546K(1028096K), 0.0218812 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T18:03:16.721-0500: 285558.898: [GC (Allocation Failure) 2018-04-16T18:03:16.721-0500: 285558.898: [ParNew: 166206K->2429K(184320K), 0.0230883 secs] 256386K->92695K(1028096K), 0.0232866 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-16T18:07:03.721-0500: 285785.898: [GC (Allocation Failure) 2018-04-16T18:07:03.721-0500: 285785.898: [ParNew: 166269K->2406K(184320K), 0.0225811 secs] 256535K->92757K(1028096K), 0.0227898 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-16T18:11:16.861-0500: 286039.039: [GC (Allocation Failure) 2018-04-16T18:11:16.861-0500: 286039.039: [ParNew: 166246K->3706K(184320K), 0.0214591 secs] 256597K->94141K(1028096K), 0.0216506 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-16T18:15:20.917-0500: 286283.094: [GC (Allocation Failure) 2018-04-16T18:15:20.917-0500: 286283.095: [ParNew: 167546K->3743K(184320K), 0.0225355 secs] 257981K->94263K(1028096K), 0.0227371 secs] [Times: user=0.07 sys=0.00, real=0.03 secs]
2018-04-16T18:18:39.537-0500: 286481.714: [GC (Allocation Failure) 2018-04-16T18:18:39.537-0500: 286481.715: [ParNew: 167583K->4072K(184320K), 0.0142604 secs] 258103K->94677K(1028096K), 0.0144509 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2018-04-16T18:22:06.883-0500: 286689.061: [GC (Allocation Failure) 2018-04-16T18:22:06.883-0500: 286689.061: [ParNew: 167912K->3967K(184320K), 0.0218256 secs] 258517K->94655K(1028096K), 0.0220566 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T18:25:31.249-0500: 286893.426: [GC (Allocation Failure) 2018-04-16T18:25:31.249-0500: 286893.426: [ParNew: 167807K->4186K(184320K), 0.0215572 secs] 258495K->94961K(1028096K), 0.0217472 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T18:29:03.213-0500: 287105.390: [GC (Allocation Failure) 2018-04-16T18:29:03.213-0500: 287105.390: [ParNew: 168026K->4134K(184320K), 0.0204636 secs] 258801K->95059K(1028096K), 0.0206507 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-16T18:32:43.408-0500: 287325.585: [GC (Allocation Failure) 2018-04-16T18:32:43.408-0500: 287325.585: [ParNew: 167974K->3060K(184320K), 0.0192851 secs] 258899K->95092K(1028096K), 0.0194765 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T18:36:15.913-0500: 287538.090: [GC (Allocation Failure) 2018-04-16T18:36:15.913-0500: 287538.091: [ParNew: 166900K->3106K(184320K), 0.0138653 secs] 258932K->95290K(1028096K), 0.0140473 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-16T18:39:44.994-0500: 287747.172: [GC (Allocation Failure) 2018-04-16T18:39:44.994-0500: 287747.172: [ParNew: 166946K->2617K(184320K), 0.0171498 secs] 259130K->95008K(1028096K), 0.0173916 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
2018-04-16T18:43:16.590-0500: 287958.768: [GC (Allocation Failure) 2018-04-16T18:43:16.590-0500: 287958.768: [ParNew: 166457K->2630K(184320K), 0.0207671 secs] 258848K->95174K(1028096K), 0.0210024 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T18:46:52.075-0500: 288174.252: [GC (Allocation Failure) 2018-04-16T18:46:52.075-0500: 288174.253: [ParNew: 166470K->2873K(184320K), 0.0202695 secs] 259014K->95567K(1028096K), 0.0204588 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2018-04-16T18:50:29.155-0500: 288391.333: [GC (Allocation Failure) 2018-04-16T18:50:29.155-0500: 288391.333: [ParNew: 166713K->2654K(184320K), 0.0205306 secs] 259407K->95435K(1028096K), 0.0207212 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T18:54:01.686-0500: 288603.863: [GC (Allocation Failure) 2018-04-16T18:54:01.686-0500: 288603.864: [ParNew: 166494K->2967K(184320K), 0.0204847 secs] 259275K->95833K(1028096K), 0.0206738 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T18:57:38.454-0500: 288820.631: [GC (Allocation Failure) 2018-04-16T18:57:38.454-0500: 288820.631: [ParNew: 166807K->2976K(184320K), 0.0202841 secs] 259673K->95927K(1028096K), 0.0204928 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T19:01:13.664-0500: 289035.842: [GC (Allocation Failure) 2018-04-16T19:01:13.664-0500: 289035.842: [ParNew: 166816K->2669K(184320K), 0.0190484 secs] 259767K->95704K(1028096K), 0.0192383 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2018-04-16T19:04:54.478-0500: 289256.655: [GC (Allocation Failure) 2018-04-16T19:04:54.478-0500: 289256.655: [ParNew: 166509K->2863K(184320K), 0.0200899 secs] 259544K->95985K(1028096K), 0.0202767 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T19:08:22.458-0500: 289464.635: [GC (Allocation Failure) 2018-04-16T19:08:22.458-0500: 289464.635: [ParNew: 166703K->2784K(184320K), 0.0200984 secs] 259825K->96058K(1028096K), 0.0202853 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T19:11:49.538-0500: 289671.715: [GC (Allocation Failure) 2018-04-16T19:11:49.538-0500: 289671.715: [ParNew: 166624K->2499K(184320K), 0.0196230 secs] 259898K->95990K(1028096K), 0.0198143 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T19:15:17.688-0500: 289879.866: [GC (Allocation Failure) 2018-04-16T19:15:17.688-0500: 289879.866: [ParNew: 166339K->2561K(184320K), 0.0173884 secs] 259830K->96174K(1028096K), 0.0175844 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2018-04-16T19:18:46.547-0500: 290088.724: [GC (Allocation Failure) 2018-04-16T19:18:46.547-0500: 290088.724: [ParNew: 166401K->2692K(184320K), 0.0208427 secs] 260014K->96391K(1028096K), 0.0210359 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T19:22:31.092-0500: 290313.269: [GC (Allocation Failure) 2018-04-16T19:22:31.092-0500: 290313.270: [ParNew: 166532K->2754K(184320K), 0.0199696 secs] 260231K->96533K(1028096K), 0.0201649 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-16T19:26:09.923-0500: 290532.100: [GC (Allocation Failure) 2018-04-16T19:26:09.923-0500: 290532.100: [ParNew: 166594K->3074K(184320K), 0.0169213 secs] 260373K->96939K(1028096K), 0.0171039 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
Heap
par new generation total 184320K, used 104335K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 61% used [0x00000000c0000000, 0x00000000c62e3698, 0x00000000ca000000)
from space 20480K, 15% used [0x00000000cb400000, 0x00000000cb7008b8, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 93865K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 46705K, capacity 47428K, committed 47504K, reserved 1091584K
class space used 4881K, capacity 5044K, committed 5088K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-nodo1.root.com.mx.out <==
ulimit -a for user hdfs
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.out.3 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804021337 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(2080552k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T13:37:33.465-0500: 1.836: [GC (Allocation Failure) 2018-04-02T13:37:33.465-0500: 1.836: [ParNew: 163840K->13460K(184320K), 0.0187739 secs] 163840K->13460K(1028096K), 0.0188840 secs] [Times: user=0.05 sys=0.01, real=0.02 secs]
2018-04-02T13:37:35.484-0500: 3.854: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 122916K(1028096K), 0.0142860 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-02T13:37:35.498-0500: 3.869: [CMS-concurrent-mark-start]
2018-04-02T13:37:35.504-0500: 3.874: [CMS-concurrent-mark: 0.005/0.005 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-02T13:37:35.504-0500: 3.874: [CMS-concurrent-preclean-start]
2018-04-02T13:37:35.506-0500: 3.876: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T13:37:35.506-0500: 3.876: [CMS-concurrent-abortable-preclean-start]
2018-04-02T13:37:38.281-0500: 6.652: [GC (Allocation Failure) 2018-04-02T13:37:38.281-0500: 6.652: [ParNew: 177300K->13275K(184320K), 0.0403379 secs] 177300K->17699K(1028096K), 0.0404404 secs] [Times: user=0.14 sys=0.01, real=0.04 secs]
CMS: abort preclean due to time 2018-04-02T13:37:40.571-0500: 8.941: [CMS-concurrent-abortable-preclean: 2.102/5.065 secs] [Times: user=4.83 sys=0.10, real=5.06 secs]
2018-04-02T13:37:40.571-0500: 8.942: [GC (CMS Final Remark) [YG occupancy: 34995 K (184320 K)]2018-04-02T13:37:40.571-0500: 8.942: [Rescan (parallel) , 0.0046859 secs]2018-04-02T13:37:40.576-0500: 8.947: [weak refs processing, 0.0000290 secs]2018-04-02T13:37:40.576-0500: 8.947: [class unloading, 0.0047214 secs]2018-04-02T13:37:40.581-0500: 8.951: [scrub symbol table, 0.0062649 secs]2018-04-02T13:37:40.587-0500: 8.958: [scrub string table, 0.0006669 secs][1 CMS-remark: 4424K(843776K)] 39419K(1028096K), 0.0175081 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-02T13:37:40.589-0500: 8.960: [CMS-concurrent-sweep-start]
2018-04-02T13:37:40.592-0500: 8.962: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2018-04-02T13:37:40.592-0500: 8.962: [CMS-concurrent-reset-start]
2018-04-02T13:37:40.599-0500: 8.970: [CMS-concurrent-reset: 0.007/0.007 secs] [Times: user=0.00 sys=0.01, real=0.00 secs]
2018-04-02T13:51:04.048-0500: 812.419: [GC (Allocation Failure) 2018-04-02T13:51:04.048-0500: 812.419: [ParNew: 177115K->10388K(184320K), 0.0210216 secs] 181539K->20037K(1028096K), 0.0212019 secs] [Times: user=0.06 sys=0.01, real=0.02 secs]
2018-04-02T14:01:15.398-0500: 1423.768: [GC (Allocation Failure) 2018-04-02T14:01:15.398-0500: 1423.768: [ParNew: 174228K->11497K(184320K), 0.0183698 secs] 183877K->21146K(1028096K), 0.0185599 secs] [Times: user=0.05 sys=0.01, real=0.02 secs]
2018-04-02T14:37:38.633-0500: 3607.004: [GC (Allocation Failure) 2018-04-02T14:37:38.633-0500: 3607.004: [ParNew: 175337K->3229K(184320K), 0.0156534 secs] 184986K->14407K(1028096K), 0.0158312 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-02T15:13:48.408-0500: 5776.779: [GC (Allocation Failure) 2018-04-02T15:13:48.408-0500: 5776.779: [ParNew: 167069K->1905K(184320K), 0.0091345 secs] 178247K->13082K(1028096K), 0.0093042 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-02T15:37:57.115-0500: 7225.486: [GC (Allocation Failure) 2018-04-02T15:37:57.115-0500: 7225.486: [ParNew: 165653K->12371K(184320K), 0.0173899 secs] 176831K->23548K(1028096K), 0.0175869 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2018-04-02T15:57:53.901-0500: 8422.272: [GC (Allocation Failure) 2018-04-02T15:57:53.902-0500: 8422.272: [ParNew: 176211K->10401K(184320K), 0.0174500 secs] 187388K->22765K(1028096K), 0.0176641 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 85350K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 45% used [0x00000000c0000000, 0x00000000c4931450, 0x00000000ca000000)
from space 20480K, 50% used [0x00000000ca000000, 0x00000000caa284e0, 0x00000000cb400000)
to space 20480K, 0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
concurrent mark-sweep generation total 843776K, used 12363K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 44508K, capacity 45026K, committed 45340K, reserved 1089536K
class space used 4698K, capacity 4879K, committed 4980K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-journalnode-nodo1.root.com.mx.out.1 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.10 <==
2018-04-04 11:58:20,975 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172899_1432085, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:21,067 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172900_1432086 src: /10.1.5.21:48346 dest: /10.1.5.20:50010
2018-04-04 11:58:21,069 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48346, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172900_1432086, duration: 911933
2018-04-04 11:58:21,069 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172900_1432086, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:21,170 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172901_1432087 src: /10.1.5.21:48356 dest: /10.1.5.20:50010
2018-04-04 11:58:21,172 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48356, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172901_1432087, duration: 806924
2018-04-04 11:58:21,172 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172901_1432087, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:21,266 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172902_1432088 src: /10.1.5.21:48360 dest: /10.1.5.20:50010
2018-04-04 11:58:21,268 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48360, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172902_1432088, duration: 889996
2018-04-04 11:58:21,268 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172902_1432088, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:21,360 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172903_1432089 src: /10.1.5.21:48366 dest: /10.1.5.20:50010
2018-04-04 11:58:21,362 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48366, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172903_1432089, duration: 839901
2018-04-04 11:58:21,362 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172903_1432089, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:21,456 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172904_1432090 src: /10.1.5.21:48370 dest: /10.1.5.20:50010
2018-04-04 11:58:21,458 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48370, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172904_1432090, duration: 1019348
2018-04-04 11:58:21,458 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172904_1432090, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:21,877 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172905_1432091 src: /10.1.5.21:48374 dest: /10.1.5.20:50010
2018-04-04 11:58:21,879 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48374, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172905_1432091, duration: 927445
2018-04-04 11:58:21,879 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172905_1432091, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:21,971 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172906_1432092 src: /10.1.5.21:48378 dest: /10.1.5.20:50010
2018-04-04 11:58:21,973 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48378, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172906_1432092, duration: 892111
2018-04-04 11:58:21,973 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172906_1432092, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:22,067 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172907_1432093 src: /10.1.5.21:48382 dest: /10.1.5.20:50010
2018-04-04 11:58:22,069 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48382, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172907_1432093, duration: 818245
2018-04-04 11:58:22,069 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172907_1432093, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:22,163 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172908_1432094 src: /10.1.5.21:48386 dest: /10.1.5.20:50010
2018-04-04 11:58:22,165 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48386, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172908_1432094, duration: 967108
2018-04-04 11:58:22,165 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172908_1432094, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:22,258 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172909_1432095 src: /10.1.5.21:48392 dest: /10.1.5.20:50010
2018-04-04 11:58:22,260 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48392, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172909_1432095, duration: 1027591
2018-04-04 11:58:22,260 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172909_1432095, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:22,354 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172910_1432096 src: /10.1.5.21:48396 dest: /10.1.5.20:50010
2018-04-04 11:58:22,356 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48396, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172910_1432096, duration: 1141370
2018-04-04 11:58:22,356 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172910_1432096, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:22,448 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172911_1432097 src: /10.1.5.21:48400 dest: /10.1.5.20:50010
2018-04-04 11:58:22,450 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48400, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172911_1432097, duration: 1023708
2018-04-04 11:58:22,450 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172911_1432097, type=LAST_IN_PIPELINE terminating
2018-04-04 11:58:22,544 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075172912_1432098 src: /10.1.5.21:48404 dest: /10.1.5.20:50010
2018-04-04 11:58:22,546 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:48404, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1733896293_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075172912_1432098, duration: 928704
2018-04-04 11:58:22,546 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075172912_1432098, type=LAST_IN_PIPELINE terminating
==> /var/log/hadoop/hdfs/gc.log-201804021108 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(6629880k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T11:09:00.112-0500: 1.952: [GC (Allocation Failure) 2018-04-02T11:09:00.112-0500: 1.952: [ParNew: 163840K->13899K(184320K), 0.0164816 secs] 163840K->13899K(1028096K), 0.0166098 secs] [Times: user=0.05 sys=0.02, real=0.01 secs]
2018-04-02T11:09:02.128-0500: 3.968: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 118013K(1028096K), 0.0175841 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-02T11:09:02.146-0500: 3.986: [CMS-concurrent-mark-start]
2018-04-02T11:09:02.158-0500: 3.998: [CMS-concurrent-mark: 0.011/0.011 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
2018-04-02T11:09:02.158-0500: 3.998: [CMS-concurrent-preclean-start]
2018-04-02T11:09:02.161-0500: 4.001: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-02T11:09:02.162-0500: 4.001: [CMS-concurrent-abortable-preclean-start]
2018-04-02T11:09:05.429-0500: 7.269: [GC (Allocation Failure) 2018-04-02T11:09:05.430-0500: 7.270: [ParNew: 177739K->13714K(184320K), 0.0426036 secs] 177739K->18150K(1028096K), 0.0427704 secs] [Times: user=0.13 sys=0.02, real=0.05 secs]
CMS: abort preclean due to time 2018-04-02T11:09:07.201-0500: 9.041: [CMS-concurrent-abortable-preclean: 2.171/5.039 secs] [Times: user=4.37 sys=0.07, real=5.03 secs]
2018-04-02T11:09:07.201-0500: 9.041: [GC (CMS Final Remark) [YG occupancy: 17471 K (184320 K)]2018-04-02T11:09:07.201-0500: 9.041: [Rescan (parallel) , 0.0033220 secs]2018-04-02T11:09:07.205-0500: 9.045: [weak refs processing, 0.0000411 secs]2018-04-02T11:09:07.205-0500: 9.045: [class unloading, 0.0070366 secs]2018-04-02T11:09:07.212-0500: 9.052: [scrub symbol table, 0.0088343 secs]2018-04-02T11:09:07.221-0500: 9.061: [scrub string table, 0.0012416 secs][1 CMS-remark: 4436K(843776K)] 21907K(1028096K), 0.0219912 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2018-04-02T11:09:07.223-0500: 9.063: [CMS-concurrent-sweep-start]
2018-04-02T11:09:07.227-0500: 9.067: [CMS-concurrent-sweep: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T11:09:07.227-0500: 9.067: [CMS-concurrent-reset-start]
2018-04-02T11:09:07.237-0500: 9.077: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
2018-04-02T11:24:41.831-0500: 943.671: [GC (Allocation Failure) 2018-04-02T11:24:41.831-0500: 943.671: [ParNew: 177484K->8691K(184320K), 0.0230098 secs] 181920K->18439K(1028096K), 0.0231917 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
Heap
par new generation total 184320K, used 109165K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 61% used [0x00000000c0000000, 0x00000000c621e880, 0x00000000ca000000)
from space 20480K, 42% used [0x00000000cb400000, 0x00000000cbc7cd68, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 9748K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 37189K, capacity 37738K, committed 38044K, reserved 1083392K
class space used 4101K, capacity 4267K, committed 4340K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804021259 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(3193336k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T12:59:08.395-0500: 1.907: [GC (Allocation Failure) 2018-04-02T12:59:08.396-0500: 1.907: [ParNew: 163840K->13426K(184320K), 0.0181485 secs] 163840K->13426K(1028096K), 0.0182661 secs] [Times: user=0.06 sys=0.01, real=0.02 secs]
2018-04-02T12:59:10.414-0500: 3.925: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 122816K(1028096K), 0.0168068 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-02T12:59:10.431-0500: 3.942: [CMS-concurrent-mark-start]
2018-04-02T12:59:10.442-0500: 3.954: [CMS-concurrent-mark: 0.011/0.011 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-02T12:59:10.442-0500: 3.954: [CMS-concurrent-preclean-start]
2018-04-02T12:59:10.446-0500: 3.958: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T12:59:10.446-0500: 3.958: [CMS-concurrent-abortable-preclean-start]
2018-04-02T12:59:14.407-0500: 7.919: [GC (Allocation Failure) 2018-04-02T12:59:14.408-0500: 7.919: [ParNew: 177266K->13560K(184320K), 0.0410668 secs] 177266K->17950K(1028096K), 0.0411927 secs] [Times: user=0.11 sys=0.02, real=0.04 secs]
CMS: abort preclean due to time 2018-04-02T12:59:15.545-0500: 9.056: [CMS-concurrent-abortable-preclean: 2.209/5.099 secs] [Times: user=4.23 sys=0.07, real=5.10 secs]
2018-04-02T12:59:15.545-0500: 9.057: [GC (CMS Final Remark) [YG occupancy: 23128 K (184320 K)]2018-04-02T12:59:15.545-0500: 9.057: [Rescan (parallel) , 0.0030870 secs]2018-04-02T12:59:15.549-0500: 9.060: [weak refs processing, 0.0000485 secs]2018-04-02T12:59:15.549-0500: 9.060: [class unloading, 0.0044492 secs]2018-04-02T12:59:15.553-0500: 9.064: [scrub symbol table, 0.0055503 secs]2018-04-02T12:59:15.559-0500: 9.070: [scrub string table, 0.0006061 secs][1 CMS-remark: 4390K(843776K)] 27518K(1028096K), 0.0145261 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-02T12:59:15.560-0500: 9.071: [CMS-concurrent-sweep-start]
2018-04-02T12:59:15.563-0500: 9.074: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2018-04-02T12:59:15.563-0500: 9.074: [CMS-concurrent-reset-start]
2018-04-02T12:59:15.570-0500: 9.081: [CMS-concurrent-reset: 0.007/0.007 secs] [Times: user=0.00 sys=0.01, real=0.00 secs]
2018-04-02T13:06:09.982-0500: 423.493: [GC (Allocation Failure) 2018-04-02T13:06:09.982-0500: 423.493: [ParNew: 177400K->14962K(184320K), 0.0283113 secs] 181790K->24697K(1028096K), 0.0284880 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-02T13:06:53.124-0500: 466.635: [GC (Allocation Failure) 2018-04-02T13:06:53.124-0500: 466.636: [ParNew: 178802K->20480K(184320K), 0.0424235 secs] 188537K->41387K(1028096K), 0.0425941 secs] [Times: user=0.04 sys=0.02, real=0.04 secs]
Heap
par new generation total 184320K, used 132200K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 68% used [0x00000000c0000000, 0x00000000c6d1a3a0, 0x00000000ca000000)
from space 20480K, 100% used [0x00000000ca000000, 0x00000000cb400000, 0x00000000cb400000)
to space 20480K, 0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
concurrent mark-sweep generation total 843776K, used 20907K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 36332K, capacity 36808K, committed 37020K, reserved 1083392K
class space used 3959K, capacity 4098K, committed 4108K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-journalnode-nodo1.root.com.mx.out <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804251757 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(9209356k free), swap 4194300k(4141584k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T17:57:47.288-0500: 1.451: [GC (Allocation Failure) 2018-04-25T17:57:47.288-0500: 1.451: [ParNew: 104960K->11135K(118016K), 0.0157160 secs] 104960K->11135K(1035520K), 0.0158234 secs] [Times: user=0.06 sys=0.02, real=0.01 secs]
2018-04-25T17:58:48.250-0500: 62.413: [GC (Allocation Failure) 2018-04-25T17:58:48.250-0500: 62.413: [ParNew: 116095K->12774K(118016K), 0.0754500 secs] 116095K->24819K(1035520K), 0.0755761 secs] [Times: user=0.50 sys=0.02, real=0.08 secs]
2018-04-25T17:58:48.326-0500: 62.489: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12044K(917504K)] 25399K(1035520K), 0.0119178 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
2018-04-25T17:58:48.338-0500: 62.501: [CMS-concurrent-mark-start]
2018-04-25T17:58:48.348-0500: 62.511: [CMS-concurrent-mark: 0.010/0.010 secs] [Times: user=0.07 sys=0.00, real=0.01 secs]
2018-04-25T17:58:48.349-0500: 62.511: [CMS-concurrent-preclean-start]
2018-04-25T17:58:48.352-0500: 62.514: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-25T17:58:48.352-0500: 62.515: [GC (CMS Final Remark) [YG occupancy: 13354 K (118016 K)]2018-04-25T17:58:48.352-0500: 62.515: [Rescan (parallel) , 0.0105602 secs]2018-04-25T17:58:48.363-0500: 62.525: [weak refs processing, 0.0000426 secs]2018-04-25T17:58:48.363-0500: 62.525: [class unloading, 0.0027570 secs]2018-04-25T17:58:48.366-0500: 62.528: [scrub symbol table, 0.0038572 secs]2018-04-25T17:58:48.369-0500: 62.532: [scrub string table, 0.0004167 secs][1 CMS-remark: 12044K(917504K)] 25399K(1035520K), 0.0182445 secs] [Times: user=0.10 sys=0.00, real=0.02 secs]
2018-04-25T17:58:48.371-0500: 62.533: [CMS-concurrent-sweep-start]
2018-04-25T17:58:48.374-0500: 62.537: [CMS-concurrent-sweep: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-25T17:58:48.374-0500: 62.537: [CMS-concurrent-reset-start]
2018-04-25T17:58:48.380-0500: 62.542: [CMS-concurrent-reset: 0.005/0.005 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
Heap
par new generation total 118016K, used 77802K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 61% used [0x00000000c0000000, 0x00000000c3f81040, 0x00000000c6680000)
from space 13056K, 97% used [0x00000000c6680000, 0x00000000c72f9bb0, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 12039K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 24170K, capacity 24410K, committed 24652K, reserved 1071104K
class space used 2668K, capacity 2778K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804251814 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(14525188k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T18:14:33.099-0500: 5.942: [GC (Allocation Failure) 2018-04-25T18:14:33.099-0500: 5.942: [ParNew: 163840K->12707K(184320K), 0.0169157 secs] 163840K->12707K(1028096K), 0.0170892 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
2018-04-25T18:14:35.116-0500: 7.959: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 51004K(1028096K), 0.0048138 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T18:14:35.121-0500: 7.964: [CMS-concurrent-mark-start]
2018-04-25T18:14:35.126-0500: 7.969: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-25T18:14:35.126-0500: 7.969: [CMS-concurrent-preclean-start]
2018-04-25T18:14:35.127-0500: 7.970: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-25T18:14:35.127-0500: 7.970: [CMS-concurrent-abortable-preclean-start]
Heap
par new generation total 184320K, used 148565K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 82% used [0x00000000c0000000, 0x00000000c84ac9f8, 0x00000000ca000000)
from space 20480K, 62% used [0x00000000cb400000, 0x00000000cc068d80, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25018K, capacity 25316K, committed 25612K, reserved 1071104K
class space used 2981K, capacity 3103K, committed 3144K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.5 <==
2018-04-14 18:52:35,961 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:49674, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177785_2436986, duration: 1881993
2018-04-14 18:52:35,962 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177785_2436986, type=LAST_IN_PIPELINE terminating
2018-04-14 18:52:36,115 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177786_2436987 src: /10.1.5.22:49676 dest: /10.1.5.20:50010
2018-04-14 18:52:36,124 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:49676, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177786_2436987, duration: 7214161
2018-04-14 18:52:36,124 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177786_2436987, type=LAST_IN_PIPELINE terminating
2018-04-14 18:52:36,313 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177787_2436988 src: /10.1.5.21:33102 dest: /10.1.5.20:50010
2018-04-14 18:52:36,316 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:33102, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177787_2436988, duration: 1397228
2018-04-14 18:52:36,316 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177787_2436988, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-14 18:52:36,470 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177788_2436989 src: /10.1.5.22:49678 dest: /10.1.5.20:50010
2018-04-14 18:52:36,473 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:49678, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177788_2436989, duration: 1360241
2018-04-14 18:52:36,473 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177788_2436989, type=LAST_IN_PIPELINE terminating
2018-04-14 18:52:37,011 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177789_2436990 src: /10.1.5.21:33110 dest: /10.1.5.20:50010
2018-04-14 18:52:37,014 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:33110, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177789_2436990, duration: 1707258
2018-04-14 18:52:37,015 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177789_2436990, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-14 18:52:37,186 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177790_2436991 src: /10.1.5.21:33118 dest: /10.1.5.20:50010
2018-04-14 18:52:37,190 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:33118, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177790_2436991, duration: 1906162
2018-04-14 18:52:37,190 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177790_2436991, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-14 18:52:37,352 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177791_2436992 src: /10.1.5.22:49680 dest: /10.1.5.20:50010
2018-04-14 18:52:37,355 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:49680, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177791_2436992, duration: 2066204
2018-04-14 18:52:37,355 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177791_2436992, type=LAST_IN_PIPELINE terminating
2018-04-14 18:52:37,722 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177792_2436993 src: /10.1.5.21:33126 dest: /10.1.5.20:50010
2018-04-14 18:52:37,726 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:33126, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177792_2436993, duration: 1937425
2018-04-14 18:52:37,726 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177792_2436993, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-14 18:52:37,888 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177793_2436994 src: /10.1.5.21:33130 dest: /10.1.5.20:50010
2018-04-14 18:52:37,892 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:33130, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177793_2436994, duration: 1846834
2018-04-14 18:52:37,892 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177793_2436994, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-14 18:52:38,110 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177794_2436995 src: /10.1.5.21:33134 dest: /10.1.5.20:50010
2018-04-14 18:52:38,114 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:33134, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177794_2436995, duration: 1695199
2018-04-14 18:52:38,114 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177794_2436995, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-14 18:52:38,276 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177795_2436996 src: /10.1.5.22:49684 dest: /10.1.5.20:50010
2018-04-14 18:52:38,279 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:49684, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177795_2436996, duration: 2127678
2018-04-14 18:52:38,279 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177795_2436996, type=LAST_IN_PIPELINE terminating
2018-04-14 18:52:38,903 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177796_2436997 src: /10.1.5.22:49690 dest: /10.1.5.20:50010
2018-04-14 18:52:38,905 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:49690, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177796_2436997, duration: 1468192
2018-04-14 18:52:38,906 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177796_2436997, type=LAST_IN_PIPELINE terminating
2018-04-14 18:52:39,087 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177797_2436998 src: /10.1.5.22:49692 dest: /10.1.5.20:50010
2018-04-14 18:52:39,089 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:49692, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177797_2436998, duration: 1369168
2018-04-14 18:52:39,090 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076177797_2436998, type=LAST_IN_PIPELINE terminating
2018-04-14 18:52:39,294 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076177798_2436999 src: /10.1.5.22:49694 dest: /10.1.5.20:50010
2018-04-14 18:52:39,296 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:49694, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076177798_2436999, duration: 1342942
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.4 <==
2018-04-15 05:35:48,475 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377537_2636738, type=LAST_IN_PIPELINE terminating
2018-04-15 05:35:48,704 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377538_2636739 src: /10.1.5.21:35738 dest: /10.1.5.20:50010
2018-04-15 05:35:48,708 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:35738, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377538_2636739, duration: 1825949
2018-04-15 05:35:48,708 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377538_2636739, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 05:35:48,848 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377539_2636740 src: /10.1.5.21:35742 dest: /10.1.5.20:50010
2018-04-15 05:35:48,852 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:35742, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377539_2636740, duration: 1792343
2018-04-15 05:35:48,852 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377539_2636740, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 05:35:48,993 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377540_2636741 src: /10.1.5.22:36884 dest: /10.1.5.20:50010
2018-04-15 05:35:48,996 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:36884, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377540_2636741, duration: 1647587
2018-04-15 05:35:48,996 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377540_2636741, type=LAST_IN_PIPELINE terminating
2018-04-15 05:35:49,136 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377541_2636742 src: /10.1.5.21:35752 dest: /10.1.5.20:50010
2018-04-15 05:35:49,140 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:35752, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377541_2636742, duration: 1752093
2018-04-15 05:35:49,141 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377541_2636742, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 05:35:49,287 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377542_2636743 src: /10.1.5.21:35756 dest: /10.1.5.20:50010
2018-04-15 05:35:49,291 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:35756, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377542_2636743, duration: 2048055
2018-04-15 05:35:49,291 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377542_2636743, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 05:35:49,436 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377543_2636744 src: /10.1.5.22:36888 dest: /10.1.5.20:50010
2018-04-15 05:35:49,439 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:36888, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377543_2636744, duration: 2026097
2018-04-15 05:35:49,439 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377543_2636744, type=LAST_IN_PIPELINE terminating
2018-04-15 05:35:49,579 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377544_2636745 src: /10.1.5.21:35766 dest: /10.1.5.20:50010
2018-04-15 05:35:49,583 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:35766, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377544_2636745, duration: 1909714
2018-04-15 05:35:49,583 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377544_2636745, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 05:35:50,236 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377545_2636746 src: /10.1.5.21:35772 dest: /10.1.5.20:50010
2018-04-15 05:35:50,240 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:35772, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377545_2636746, duration: 2085471
2018-04-15 05:35:50,240 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377545_2636746, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 05:35:50,380 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377546_2636747 src: /10.1.5.21:35776 dest: /10.1.5.20:50010
2018-04-15 05:35:50,384 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:35776, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377546_2636747, duration: 1998998
2018-04-15 05:35:50,384 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377546_2636747, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 05:35:50,523 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377547_2636748 src: /10.1.5.22:36890 dest: /10.1.5.20:50010
2018-04-15 05:35:50,526 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:36890, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377547_2636748, duration: 2202201
2018-04-15 05:35:50,526 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377547_2636748, type=LAST_IN_PIPELINE terminating
2018-04-15 05:35:50,666 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377548_2636749 src: /10.1.5.22:36892 dest: /10.1.5.20:50010
2018-04-15 05:35:50,669 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:36892, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377548_2636749, duration: 1776813
2018-04-15 05:35:50,669 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377548_2636749, type=LAST_IN_PIPELINE terminating
2018-04-15 05:35:50,810 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377549_2636750 src: /10.1.5.22:36894 dest: /10.1.5.20:50010
2018-04-15 05:35:50,813 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:36894, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377549_2636750, duration: 1729657
2018-04-15 05:35:50,813 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377549_2636750, type=LAST_IN_PIPELINE terminating
2018-04-15 05:35:51,025 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076377550_2636751 src: /10.1.5.21:35796 dest: /10.1.5.20:50010
2018-04-15 05:35:51,029 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:35796, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076377550_2636751, duration: 2099006
2018-04-15 05:35:51,029 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076377550_2636751, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
==> /var/log/hadoop/hdfs/gc.log-201804021135 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(6795636k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T11:35:28.988-0500: 1.914: [GC (Allocation Failure) 2018-04-02T11:35:28.988-0500: 1.914: [ParNew: 163840K->13428K(184320K), 0.0189971 secs] 163840K->13428K(1028096K), 0.0191262 secs] [Times: user=0.03 sys=0.01, real=0.02 secs]
2018-04-02T11:35:31.007-0500: 3.934: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 122876K(1028096K), 0.0161394 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-02T11:35:31.023-0500: 3.950: [CMS-concurrent-mark-start]
2018-04-02T11:35:31.030-0500: 3.957: [CMS-concurrent-mark: 0.007/0.007 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T11:35:31.030-0500: 3.957: [CMS-concurrent-preclean-start]
2018-04-02T11:35:31.033-0500: 3.959: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2018-04-02T11:35:31.033-0500: 3.959: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2018-04-02T11:35:36.080-0500: 9.006: [CMS-concurrent-abortable-preclean: 2.100/5.047 secs] [Times: user=2.10 sys=0.01, real=5.04 secs]
2018-04-02T11:35:36.080-0500: 9.006: [GC (CMS Final Remark) [YG occupancy: 128783 K (184320 K)]2018-04-02T11:35:36.080-0500: 9.006: [Rescan (parallel) , 0.0119394 secs]2018-04-02T11:35:36.092-0500: 9.018: [weak refs processing, 0.0000222 secs]2018-04-02T11:35:36.092-0500: 9.018: [class unloading, 0.0032137 secs]2018-04-02T11:35:36.095-0500: 9.022: [scrub symbol table, 0.0045934 secs]2018-04-02T11:35:36.100-0500: 9.026: [scrub string table, 0.0003435 secs][1 CMS-remark: 0K(843776K)] 128783K(1028096K), 0.0208070 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]
2018-04-02T11:35:36.101-0500: 9.027: [CMS-concurrent-sweep-start]
2018-04-02T11:35:36.101-0500: 9.027: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T11:35:36.101-0500: 9.027: [CMS-concurrent-reset-start]
2018-04-02T11:35:36.110-0500: 9.036: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.00 sys=0.01, real=0.00 secs]
2018-04-02T11:35:38.546-0500: 11.473: [GC (Allocation Failure) 2018-04-02T11:35:38.546-0500: 11.473: [ParNew: 177268K->14668K(184320K), 0.0488528 secs] 177268K->19060K(1028096K), 0.0489795 secs] [Times: user=0.16 sys=0.01, real=0.05 secs]
2018-04-02T11:46:17.251-0500: 650.177: [GC (Allocation Failure) 2018-04-02T11:46:17.251-0500: 650.178: [ParNew: 178508K->10743K(184320K), 0.0161668 secs] 182900K->20495K(1028096K), 0.0163743 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
2018-04-02T11:46:18.041-0500: 650.967: [GC (Allocation Failure) 2018-04-02T11:46:18.041-0500: 650.967: [ParNew: 174583K->9852K(184320K), 0.0073793 secs] 184335K->19604K(1028096K), 0.0074757 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-02T11:46:18.687-0500: 651.614: [GC (Allocation Failure) 2018-04-02T11:46:18.687-0500: 651.614: [ParNew: 173692K->7444K(184320K), 0.0074947 secs] 183444K->17196K(1028096K), 0.0076370 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-02T11:46:21.171-0500: 654.098: [GC (Allocation Failure) 2018-04-02T11:46:21.172-0500: 654.098: [ParNew: 171284K->8335K(184320K), 0.0079606 secs] 181036K->18087K(1028096K), 0.0081394 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-02T11:53:59.232-0500: 1112.159: [GC (Allocation Failure) 2018-04-02T11:53:59.232-0500: 1112.159: [ParNew: 172175K->3549K(184320K), 0.0103064 secs] 181927K->13301K(1028096K), 0.0105127 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-02T12:27:58.919-0500: 3151.845: [GC (Allocation Failure) 2018-04-02T12:27:58.919-0500: 3151.845: [ParNew: 167389K->5519K(184320K), 0.0132838 secs] 177141K->15271K(1028096K), 0.0134568 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
Heap
par new generation total 184320K, used 142966K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 83% used [0x00000000c0000000, 0x00000000c8639b30, 0x00000000ca000000)
from space 20480K, 26% used [0x00000000ca000000, 0x00000000ca563e40, 0x00000000cb400000)
to space 20480K, 0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
concurrent mark-sweep generation total 843776K, used 9752K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 41939K, capacity 42408K, committed 42584K, reserved 1087488K
class space used 4525K, capacity 4706K, committed 4780K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804251850 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(14451276k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T18:50:14.441-0500: 1.750: [GC (Allocation Failure) 2018-04-25T18:50:14.441-0500: 1.751: [ParNew: 163840K->12695K(184320K), 0.0128244 secs] 163840K->12695K(1028096K), 0.0129442 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 148553K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 82% used [0x00000000c0000000, 0x00000000c84ac830, 0x00000000ca000000)
from space 20480K, 61% used [0x00000000cb400000, 0x00000000cc065df0, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25039K, capacity 25316K, committed 25612K, reserved 1071104K
class space used 2983K, capacity 3103K, committed 3144K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.7 <==
2018-04-13 21:04:37,110 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075778150_2037351, type=LAST_IN_PIPELINE terminating
2018-04-13 21:04:37,258 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075778151_2037352 src: /10.1.5.21:50766 dest: /10.1.5.20:50010
2018-04-13 21:04:37,262 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:50766, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075778151_2037352, duration: 1987639
2018-04-13 21:04:37,262 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075778151_2037352, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-13 21:04:37,411 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075778152_2037353 src: /10.1.5.22:43494 dest: /10.1.5.20:50010
2018-04-13 21:04:37,414 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:43494, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075778152_2037353, duration: 2093183
2018-04-13 21:04:37,414 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075778152_2037353, type=LAST_IN_PIPELINE terminating
2018-04-13 21:04:37,554 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075778153_2037354 src: /10.1.5.21:50774 dest: /10.1.5.20:50010
2018-04-13 21:04:37,558 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:50774, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075778153_2037354, duration: 1886145
2018-04-13 21:04:37,558 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075778153_2037354, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-13 21:04:37,696 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075778154_2037355 src: /10.1.5.21:50780 dest: /10.1.5.20:50010
2018-04-13 21:04:37,701 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:50780, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075778154_2037355, duration: 2011996
2018-04-13 21:04:37,701 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075778154_2037355, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-13 21:04:37,840 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075778155_2037356 src: /10.1.5.21:50784 dest: /10.1.5.20:50010
2018-04-13 21:04:37,844 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:50784, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075778155_2037356, duration: 1892047
2018-04-13 21:04:37,845 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075778155_2037356, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-13 21:04:37,985 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075778156_2037357 src: /10.1.5.22:43496 dest: /10.1.5.20:50010
2018-04-13 21:04:37,989 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:43496, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075778156_2037357, duration: 2305942
2018-04-13 21:04:37,989 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075778156_2037357, type=LAST_IN_PIPELINE terminating
2018-04-13 21:04:38,279 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1075778157_2037358 src: /10.1.5.22:43498 dest: /10.1.5.20:50010
2018-04-13 21:04:38,282 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:43498, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1075778157_2037358, duration: 2104514
2018-04-13 21:04:38,282 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1075778157_2037358, type=LAST_IN_PIPELINE terminating
2018-04-13 21:04:38,630 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778144_2037345 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778144 for deletion
2018-04-13 21:04:38,630 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778145_2037346 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778145 for deletion
2018-04-13 21:04:38,630 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778129_2037330 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778129 for deletion
2018-04-13 21:04:38,630 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075778144_2037345 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778144
2018-04-13 21:04:38,631 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778130_2037331 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778130 for deletion
2018-04-13 21:04:38,631 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075778145_2037346 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778145
2018-04-13 21:04:38,631 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778131_2037332 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778131 for deletion
2018-04-13 21:04:38,631 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075778129_2037330 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778129
2018-04-13 21:04:38,631 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778132_2037333 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778132 for deletion
2018-04-13 21:04:38,631 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075778130_2037331 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778130
2018-04-13 21:04:38,631 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778133_2037334 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778133 for deletion
2018-04-13 21:04:38,632 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075778131_2037332 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778131
2018-04-13 21:04:38,632 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778134_2037335 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778134 for deletion
2018-04-13 21:04:38,632 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075778132_2037333 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778132
2018-04-13 21:04:38,632 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778135_2037336 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778135 for deletion
2018-04-13 21:04:38,632 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075778133_2037334 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778133
2018-04-13 21:04:38,632 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(218)) - Scheduling blk_1075778136_2037337 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778136 for deletion
2018-04-13 21:04:38,632 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-645840651-10.1.5.19-1522680031790 blk_1075778134_2037335 file /mnt/datanode/current/BP-645840651-10.1.5.19-1522680031790/current/finalized/subdir31/subdir18/blk_1075778134
==> /var/log/hadoop/hdfs/gc.log-201804051659 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(13344416k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-05T16:59:13.726-0500: 1.777: [GC (Allocation Failure) 2018-04-05T16:59:13.726-0500: 1.777: [ParNew: 163840K->14102K(184320K), 0.0170243 secs] 163840K->14102K(1028096K), 0.0171227 secs] [Times: user=0.06 sys=0.01, real=0.02 secs]
2018-04-05T16:59:15.744-0500: 3.794: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 126595K(1028096K), 0.0145481 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-05T16:59:15.758-0500: 3.809: [CMS-concurrent-mark-start]
2018-04-05T16:59:15.769-0500: 3.819: [CMS-concurrent-mark: 0.011/0.011 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-05T16:59:15.769-0500: 3.820: [CMS-concurrent-preclean-start]
2018-04-05T16:59:15.773-0500: 3.823: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-05T16:59:15.773-0500: 3.823: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2018-04-05T16:59:20.901-0500: 8.951: [CMS-concurrent-abortable-preclean: 2.191/5.128 secs] [Times: user=2.62 sys=0.01, real=5.12 secs]
2018-04-05T16:59:20.901-0500: 8.952: [GC (CMS Final Remark) [YG occupancy: 147912 K (184320 K)]2018-04-05T16:59:20.901-0500: 8.952: [Rescan (parallel) , 0.0138207 secs]2018-04-05T16:59:20.915-0500: 8.965: [weak refs processing, 0.0000379 secs]2018-04-05T16:59:20.915-0500: 8.966: [class unloading, 0.0051329 secs]2018-04-05T16:59:20.920-0500: 8.971: [scrub symbol table, 0.0065955 secs]2018-04-05T16:59:20.927-0500: 8.977: [scrub string table, 0.0006222 secs][1 CMS-remark: 0K(843776K)] 147912K(1028096K), 0.0274236 secs] [Times: user=0.07 sys=0.00, real=0.03 secs]
2018-04-05T16:59:20.929-0500: 8.979: [CMS-concurrent-sweep-start]
2018-04-05T16:59:20.929-0500: 8.979: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-05T16:59:20.929-0500: 8.979: [CMS-concurrent-reset-start]
2018-04-05T16:59:20.933-0500: 8.984: [CMS-concurrent-reset: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2018-04-05T16:59:21.265-0500: 9.315: [GC (Allocation Failure) 2018-04-05T16:59:21.265-0500: 9.315: [ParNew: 177942K->11739K(184320K), 0.0417126 secs] 177942K->16207K(1028096K), 0.0418370 secs] [Times: user=0.16 sys=0.01, real=0.04 secs]
2018-04-05T17:07:54.230-0500: 522.280: [GC (Allocation Failure) 2018-04-05T17:07:54.230-0500: 522.281: [ParNew: 175579K->8515K(184320K), 0.0221007 secs] 180047K->18037K(1028096K), 0.0223002 secs] [Times: user=0.07 sys=0.01, real=0.03 secs]
Heap
par new generation total 184320K, used 168387K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 97% used [0x00000000c0000000, 0x00000000c9c1ff18, 0x00000000ca000000)
from space 20480K, 41% used [0x00000000cb400000, 0x00000000cbc50d08, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 9522K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 34402K, capacity 34830K, committed 35080K, reserved 1081344K
class space used 3781K, capacity 3921K, committed 3936K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.out.2 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804251756 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(9211156k free), swap 4194300k(4141584k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T17:56:17.688-0500: 1.752: [GC (Allocation Failure) 2018-04-25T17:56:17.688-0500: 1.752: [ParNew: 163840K->12699K(184320K), 0.0120326 secs] 163840K->12699K(1028096K), 0.0121359 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
Heap
par new generation total 184320K, used 148564K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 82% used [0x00000000c0000000, 0x00000000c84ae410, 0x00000000ca000000)
from space 20480K, 62% used [0x00000000cb400000, 0x00000000cc066f68, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25010K, capacity 25312K, committed 25612K, reserved 1071104K
class space used 2981K, capacity 3103K, committed 3144K, reserved 1048576K
==> /var/log/hadoop/hdfs/SecurityAuth.audit <==
2018-04-23 13:01:13,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:01:28,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:01:28,940 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:01:58,938 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:02:13,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:02:28,933 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:02:43,941 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:02:58,940 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:03:13,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:03:13,941 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:03:43,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:03:43,940 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:03:58,937 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:04:13,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:07:28,936 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:07:28,944 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:07:43,935 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:07:43,941 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:08:28,935 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:08:28,941 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:08:43,941 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:08:58,935 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:09:13,935 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:09:28,942 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:09:58,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:10:13,941 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:10:28,941 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:10:43,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:10:43,940 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:10:58,932 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:10:58,938 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:11:13,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:11:13,941 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:11:28,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:11:28,940 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:11:58,939 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:12:28,934 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:12:28,940 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:12:43,940 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
2018-04-23 13:13:13,933 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for yarn (auth:SIMPLE)
==> /var/log/hadoop/hdfs/hdfs-audit.log <==
==> /var/log/hadoop/hdfs/gc.log-201804061609 <==
2018-04-09T16:45:36.368-0500: 261370.908: [GC (Allocation Failure) 2018-04-09T16:45:36.368-0500: 261370.908: [ParNew: 171426K->4659K(184320K), 0.0068890 secs] 184506K->17739K(1028096K), 0.0070210 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T16:46:57.844-0500: 261452.385: [GC (Allocation Failure) 2018-04-09T16:46:57.844-0500: 261452.385: [ParNew: 168499K->4082K(184320K), 0.0069807 secs] 181579K->17273K(1028096K), 0.0071462 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T16:48:21.065-0500: 261535.605: [GC (Allocation Failure) 2018-04-09T16:48:21.065-0500: 261535.605: [ParNew: 167922K->4341K(184320K), 0.0078561 secs] 181113K->17759K(1028096K), 0.0079600 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T16:49:37.412-0500: 261611.952: [GC (Allocation Failure) 2018-04-09T16:49:37.412-0500: 261611.952: [ParNew: 168181K->3097K(184320K), 0.0055818 secs] 181599K->16516K(1028096K), 0.0057533 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-09T16:51:08.178-0500: 261702.719: [GC (Allocation Failure) 2018-04-09T16:51:08.178-0500: 261702.719: [ParNew: 166937K->3607K(184320K), 0.0059845 secs] 180356K->17037K(1028096K), 0.0061777 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T17:02:40.719-0500: 262395.260: [GC (Allocation Failure) 2018-04-09T17:02:40.720-0500: 262395.260: [ParNew: 167447K->9186K(184320K), 0.0125949 secs] 180877K->22716K(1028096K), 0.0127747 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T17:04:04.963-0500: 262479.504: [GC (Allocation Failure) 2018-04-09T17:04:04.963-0500: 262479.504: [ParNew: 173026K->4154K(184320K), 0.0081837 secs] 186556K->17843K(1028096K), 0.0083774 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T17:05:16.893-0500: 262551.433: [GC (Allocation Failure) 2018-04-09T17:05:16.893-0500: 262551.433: [ParNew: 167994K->3847K(184320K), 0.0062962 secs] 181683K->17537K(1028096K), 0.0065091 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T17:06:59.773-0500: 262654.313: [GC (Allocation Failure) 2018-04-09T17:06:59.773-0500: 262654.313: [ParNew: 167687K->3537K(184320K), 0.0080876 secs] 181377K->17227K(1028096K), 0.0082548 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T17:08:45.617-0500: 262760.157: [GC (Allocation Failure) 2018-04-09T17:08:45.617-0500: 262760.157: [ParNew: 167377K->3133K(184320K), 0.0076898 secs] 181067K->16844K(1028096K), 0.0078811 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T17:13:57.412-0500: 263071.952: [GC (Allocation Failure) 2018-04-09T17:13:57.412-0500: 263071.952: [ParNew: 166973K->3274K(184320K), 0.0071363 secs] 180684K->16984K(1028096K), 0.0073205 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T18:01:17.411-0500: 265911.952: [GC (Allocation Failure) 2018-04-09T18:01:17.412-0500: 265911.952: [ParNew: 167114K->990K(184320K), 0.0074652 secs] 180824K->14707K(1028096K), 0.0076482 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T18:47:37.416-0500: 268691.956: [GC (Allocation Failure) 2018-04-09T18:47:37.416-0500: 268691.956: [ParNew: 164830K->1151K(184320K), 0.0075219 secs] 178547K->14873K(1028096K), 0.0076947 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T19:34:12.179-0500: 271486.719: [GC (Allocation Failure) 2018-04-09T19:34:12.179-0500: 271486.719: [ParNew: 164991K->969K(184320K), 0.0075405 secs] 178713K->14691K(1028096K), 0.0077343 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-09T20:21:03.288-0500: 274297.829: [GC (Allocation Failure) 2018-04-09T20:21:03.289-0500: 274297.829: [ParNew: 164809K->1008K(184320K), 0.0071114 secs] 178531K->14731K(1028096K), 0.0072799 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-09T21:08:22.507-0500: 277137.047: [GC (Allocation Failure) 2018-04-09T21:08:22.507-0500: 277137.047: [ParNew: 164848K->789K(184320K), 0.0071449 secs] 178571K->14512K(1028096K), 0.0073342 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-09T21:55:31.652-0500: 279966.192: [GC (Allocation Failure) 2018-04-09T21:55:31.652-0500: 279966.192: [ParNew: 164629K->930K(184320K), 0.0070723 secs] 178352K->14654K(1028096K), 0.0072705 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T22:43:37.415-0500: 282851.955: [GC (Allocation Failure) 2018-04-09T22:43:37.415-0500: 282851.955: [ParNew: 164770K->822K(184320K), 0.0059132 secs] 178494K->14617K(1028096K), 0.0060844 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-09T23:21:47.418-0500: 285141.958: [GC (Allocation Failure) 2018-04-09T23:21:47.418-0500: 285141.959: [ParNew: 164662K->765K(184320K), 0.0066362 secs] 178457K->14561K(1028096K), 0.0068356 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-10T00:08:22.508-0500: 287937.048: [GC (Allocation Failure) 2018-04-10T00:08:22.508-0500: 287937.048: [ParNew: 164605K->977K(184320K), 0.0073525 secs] 178401K->14774K(1028096K), 0.0075282 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-10T00:55:18.274-0500: 290752.815: [GC (Allocation Failure) 2018-04-10T00:55:18.275-0500: 290752.815: [ParNew: 164817K->878K(184320K), 0.0075870 secs] 178614K->14675K(1028096K), 0.0078008 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-10T01:42:58.683-0500: 293613.224: [GC (Allocation Failure) 2018-04-10T01:42:58.683-0500: 293613.224: [ParNew: 164718K->843K(184320K), 0.0072956 secs] 178515K->14640K(1028096K), 0.0074728 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-10T02:30:54.987-0500: 296489.528: [GC (Allocation Failure) 2018-04-10T02:30:54.988-0500: 296489.528: [ParNew: 164683K->892K(184320K), 0.0057823 secs] 178480K->14689K(1028096K), 0.0059494 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-10T03:18:29.893-0500: 299344.433: [GC (Allocation Failure) 2018-04-10T03:18:29.893-0500: 299344.433: [ParNew: 164732K->934K(184320K), 0.0035418 secs] 178529K->14730K(1028096K), 0.0036588 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-10T04:06:37.316-0500: 302231.856: [GC (Allocation Failure) 2018-04-10T04:06:37.316-0500: 302231.856: [ParNew: 164774K->964K(184320K), 0.0075358 secs] 178570K->14761K(1028096K), 0.0077377 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-10T04:49:19.858-0500: 304794.398: [GC (Allocation Failure) 2018-04-10T04:49:19.858-0500: 304794.398: [ParNew: 164804K->1821K(184320K), 0.0070126 secs] 178601K->15618K(1028096K), 0.0071673 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-10T05:31:27.411-0500: 307321.952: [GC (Allocation Failure) 2018-04-10T05:31:27.411-0500: 307321.952: [ParNew: 165661K->1503K(184320K), 0.0074752 secs] 179458K->15301K(1028096K), 0.0076847 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-10T06:19:22.551-0500: 310197.092: [GC (Allocation Failure) 2018-04-10T06:19:22.552-0500: 310197.092: [ParNew: 165343K->944K(184320K), 0.0072000 secs] 179141K->14742K(1028096K), 0.0073644 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-10T07:07:22.554-0500: 313077.094: [GC (Allocation Failure) 2018-04-10T07:07:22.554-0500: 313077.094: [ParNew: 164784K->920K(184320K), 0.0075758 secs] 178582K->14718K(1028096K), 0.0077700 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-10T07:54:32.412-0500: 315906.953: [GC (Allocation Failure) 2018-04-10T07:54:32.413-0500: 315906.953: [ParNew: 164760K->925K(184320K), 0.0048309 secs] 178558K->14723K(1028096K), 0.0049819 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-10T08:41:21.754-0500: 318716.294: [GC (Allocation Failure) 2018-04-10T08:41:21.754-0500: 318716.294: [ParNew: 164765K->1002K(184320K), 0.0072736 secs] 178563K->14800K(1028096K), 0.0074439 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-10T09:28:20.075-0500: 321534.615: [GC (Allocation Failure) 2018-04-10T09:28:20.075-0500: 321534.615: [ParNew: 164842K->887K(184320K), 0.0070417 secs] 178640K->14686K(1028096K), 0.0071903 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 124541K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 75% used [0x00000000c0000000, 0x00000000c78c15a8, 0x00000000ca000000)
from space 20480K, 4% used [0x00000000cb400000, 0x00000000cb4dde58, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 13799K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 42731K, capacity 43530K, committed 43804K, reserved 1089536K
class space used 4484K, capacity 4691K, committed 4724K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804251733 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(9033212k free), swap 4194300k(4141584k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T17:33:43.566-0500: 1.690: [GC (Allocation Failure) 2018-04-25T17:33:43.566-0500: 1.690: [ParNew: 163840K->12694K(184320K), 0.0149005 secs] 163840K->12694K(1028096K), 0.0150037 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 149373K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 83% used [0x00000000c0000000, 0x00000000c8579de0, 0x00000000ca000000)
from space 20480K, 61% used [0x00000000cb400000, 0x00000000cc0658f8, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25042K, capacity 25322K, committed 25612K, reserved 1071104K
class space used 2989K, capacity 3104K, committed 3144K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.out.1 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-nodo1.root.com.mx.out.4 <==
ulimit -a for user hdfs
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804251829 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(14259328k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T18:29:56.887-0500: 1.680: [GC (Allocation Failure) 2018-04-25T18:29:56.887-0500: 1.680: [ParNew: 104960K->11170K(118016K), 0.0151869 secs] 104960K->11170K(1035520K), 0.0152895 secs] [Times: user=0.04 sys=0.01, real=0.02 secs]
2018-04-25T18:30:58.422-0500: 63.215: [GC (Allocation Failure) 2018-04-25T18:30:58.422-0500: 63.215: [ParNew: 116130K->13056K(118016K), 0.0843952 secs] 116130K->25328K(1035520K), 0.0845265 secs] [Times: user=0.60 sys=0.01, real=0.08 secs]
2018-04-25T18:30:58.507-0500: 63.300: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12272K(917504K)] 25907K(1035520K), 0.0091819 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
2018-04-25T18:30:58.516-0500: 63.309: [CMS-concurrent-mark-start]
2018-04-25T18:30:58.529-0500: 63.322: [CMS-concurrent-mark: 0.013/0.013 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2018-04-25T18:30:58.529-0500: 63.322: [CMS-concurrent-preclean-start]
2018-04-25T18:30:58.534-0500: 63.327: [CMS-concurrent-preclean: 0.005/0.005 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-25T18:30:58.535-0500: 63.328: [GC (CMS Final Remark) [YG occupancy: 13635 K (118016 K)]2018-04-25T18:30:58.535-0500: 63.328: [Rescan (parallel) , 0.0118864 secs]2018-04-25T18:30:58.547-0500: 63.340: [weak refs processing, 0.0003454 secs]2018-04-25T18:30:58.547-0500: 63.340: [class unloading, 0.0029878 secs]2018-04-25T18:30:58.550-0500: 63.343: [scrub symbol table, 0.0040382 secs]2018-04-25T18:30:58.554-0500: 63.347: [scrub string table, 0.0003961 secs][1 CMS-remark: 12272K(917504K)] 25907K(1035520K), 0.0202583 secs] [Times: user=0.11 sys=0.00, real=0.02 secs]
2018-04-25T18:30:58.555-0500: 63.348: [CMS-concurrent-sweep-start]
2018-04-25T18:30:58.559-0500: 63.352: [CMS-concurrent-sweep: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-25T18:30:58.559-0500: 63.352: [CMS-concurrent-reset-start]
2018-04-25T18:30:58.565-0500: 63.358: [CMS-concurrent-reset: 0.006/0.006 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
Heap
par new generation total 118016K, used 90754K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 74% used [0x00000000c0000000, 0x00000000c4be0b50, 0x00000000c6680000)
from space 13056K, 100% used [0x00000000c6680000, 0x00000000c7340000, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 12266K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 24215K, capacity 24542K, committed 24908K, reserved 1071104K
class space used 2669K, capacity 2780K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.out.4 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-nodo1.root.com.mx.log <==
2018-04-25 19:03:46,691 INFO namenode.TransferFsImage (TransferFsImage.java:getFileClient(414)) - Opening connection to http://master.root.com.mx:50070/imagetransfer?getimage=1&txid=20154766&storageInfo=-63:251965488:0:CID-23256935-ff35-4a77-ab11-5583e45d2604
2018-04-25 19:03:46,855 INFO namenode.TransferFsImage (TransferFsImage.java:setTimeout(461)) - Image Transfer timeout configured to 60000 milliseconds
2018-04-25 19:03:47,035 INFO namenode.TransferFsImage (TransferFsImage.java:receiveFile(592)) - Combined time for fsimage download and fsync to all disks took 0.01s. The fsimage download took 0.01s at 27333.33 KB/s. Synchronous (fsync) write to disk of /hadoop/hdfs/namesecondary/current/fsimage.ckpt_0000000000020154766 took 0.00s.
2018-04-25 19:03:47,035 INFO namenode.TransferFsImage (TransferFsImage.java:downloadImageToStorage(116)) - Downloaded file fsimage.ckpt_0000000000020154766 size 336534 bytes.
2018-04-25 19:03:47,070 INFO namenode.TransferFsImage (TransferFsImage.java:getFileClient(414)) - Opening connection to http://master.root.com.mx:50070/imagetransfer?getedit=1&startTxId=20154767&endTxId=20154767&storageInfo=-63:251965488:0:CID-23256935-ff35-4a77-ab11-5583e45d2604
2018-04-25 19:03:47,136 INFO namenode.TransferFsImage (TransferFsImage.java:receiveFile(592)) - Combined time for fsimage download and fsync to all disks took 0.01s. The fsimage download took 0.01s at 93090.91 KB/s. Synchronous (fsync) write to disk of /hadoop/hdfs/namesecondary/current/edits_tmp_0000000000020154767-0000000000020154767_0000000000003329845 took 0.00s.
2018-04-25 19:03:47,136 INFO namenode.TransferFsImage (TransferFsImage.java:downloadEditsToStorage(169)) - Downloaded file edits_tmp_0000000000020154767-0000000000020154767_0000000000003329845 size 0 bytes.
2018-04-25 19:03:47,137 INFO namenode.TransferFsImage (TransferFsImage.java:getFileClient(414)) - Opening connection to http://master.root.com.mx:50070/imagetransfer?getedit=1&startTxId=20154768&endTxId=20154768&storageInfo=-63:251965488:0:CID-23256935-ff35-4a77-ab11-5583e45d2604
2018-04-25 19:03:47,194 INFO namenode.TransferFsImage (TransferFsImage.java:receiveFile(592)) - Combined time for fsimage download and fsync to all disks took 0.01s. The fsimage download took 0.01s at 85333.33 KB/s. Synchronous (fsync) write to disk of /hadoop/hdfs/namesecondary/current/edits_tmp_0000000000020154768-0000000000020154768_0000000000003329912 took 0.00s.
2018-04-25 19:03:47,194 INFO namenode.TransferFsImage (TransferFsImage.java:downloadEditsToStorage(169)) - Downloaded file edits_tmp_0000000000020154768-0000000000020154768_0000000000003329912 size 0 bytes.
2018-04-25 19:03:47,196 INFO namenode.TransferFsImage (TransferFsImage.java:getFileClient(414)) - Opening connection to http://master.root.com.mx:50070/imagetransfer?getedit=1&startTxId=20154769&endTxId=20154769&storageInfo=-63:251965488:0:CID-23256935-ff35-4a77-ab11-5583e45d2604
2018-04-25 19:03:47,270 INFO namenode.TransferFsImage (TransferFsImage.java:receiveFile(592)) - Combined time for fsimage download and fsync to all disks took 0.01s. The fsimage download took 0.01s at 85333.33 KB/s. Synchronous (fsync) write to disk of /hadoop/hdfs/namesecondary/current/edits_tmp_0000000000020154769-0000000000020154769_0000000000003329970 took 0.00s.
2018-04-25 19:03:47,270 INFO namenode.TransferFsImage (TransferFsImage.java:downloadEditsToStorage(169)) - Downloaded file edits_tmp_0000000000020154769-0000000000020154769_0000000000003329970 size 0 bytes.
2018-04-25 19:03:47,271 INFO namenode.TransferFsImage (TransferFsImage.java:getFileClient(414)) - Opening connection to http://master.root.com.mx:50070/imagetransfer?getedit=1&startTxId=20154770&endTxId=20154772&storageInfo=-63:251965488:0:CID-23256935-ff35-4a77-ab11-5583e45d2604
2018-04-25 19:03:47,312 INFO namenode.TransferFsImage (TransferFsImage.java:receiveFile(592)) - Combined time for fsimage download and fsync to all disks took 0.00s. The fsimage download took 0.00s at 0.00 KB/s. Synchronous (fsync) write to disk of /hadoop/hdfs/namesecondary/current/edits_tmp_0000000000020154770-0000000000020154772_0000000000003330046 took 0.00s.
2018-04-25 19:03:47,312 INFO namenode.TransferFsImage (TransferFsImage.java:downloadEditsToStorage(169)) - Downloaded file edits_tmp_0000000000020154770-0000000000020154772_0000000000003330046 size 0 bytes.
2018-04-25 19:03:47,396 INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSection(257)) - Loading 3880 INodes.
2018-04-25 19:03:47,569 INFO namenode.FSImageFormatProtobuf (FSImageFormatProtobuf.java:load(184)) - Loaded FSImage in 0 seconds.
2018-04-25 19:03:47,569 INFO namenode.FSImage (FSImage.java:loadFSImage(897)) - Loaded image for txid 20154766 from /hadoop/hdfs/namesecondary/current/fsimage_0000000000020154766
2018-04-25 19:03:47,570 INFO namenode.NameCache (NameCache.java:initialized(143)) - initialized with 4 entries 292 lookups
2018-04-25 19:03:47,574 INFO namenode.Checkpointer (Checkpointer.java:rollForwardByApplyingLogs(313)) - Checkpointer about to load edits from 4 stream(s).
2018-04-25 19:03:47,578 INFO namenode.FSImage (FSImage.java:loadEdits(835)) - Reading /hadoop/hdfs/namesecondary/current/edits_0000000000020154767-0000000000020154767 expecting start txid #20154767
2018-04-25 19:03:47,578 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(142)) - Start loading edits file /hadoop/hdfs/namesecondary/current/edits_0000000000020154767-0000000000020154767
2018-04-25 19:03:47,594 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(145)) - Edits file /hadoop/hdfs/namesecondary/current/edits_0000000000020154767-0000000000020154767 of size 1048576 edits # 1 loaded in 0 seconds
2018-04-25 19:03:47,595 INFO namenode.FSImage (FSImage.java:loadEdits(835)) - Reading /hadoop/hdfs/namesecondary/current/edits_0000000000020154768-0000000000020154768 expecting start txid #20154768
2018-04-25 19:03:47,595 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(142)) - Start loading edits file /hadoop/hdfs/namesecondary/current/edits_0000000000020154768-0000000000020154768
2018-04-25 19:03:47,596 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(145)) - Edits file /hadoop/hdfs/namesecondary/current/edits_0000000000020154768-0000000000020154768 of size 1048576 edits # 1 loaded in 0 seconds
2018-04-25 19:03:47,596 INFO namenode.FSImage (FSImage.java:loadEdits(835)) - Reading /hadoop/hdfs/namesecondary/current/edits_0000000000020154769-0000000000020154769 expecting start txid #20154769
2018-04-25 19:03:47,596 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(142)) - Start loading edits file /hadoop/hdfs/namesecondary/current/edits_0000000000020154769-0000000000020154769
2018-04-25 19:03:47,598 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(145)) - Edits file /hadoop/hdfs/namesecondary/current/edits_0000000000020154769-0000000000020154769 of size 1048576 edits # 1 loaded in 0 seconds
2018-04-25 19:03:47,598 INFO namenode.FSImage (FSImage.java:loadEdits(835)) - Reading /hadoop/hdfs/namesecondary/current/edits_0000000000020154770-0000000000020154772 expecting start txid #20154770
2018-04-25 19:03:47,598 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(142)) - Start loading edits file /hadoop/hdfs/namesecondary/current/edits_0000000000020154770-0000000000020154772
2018-04-25 19:03:47,643 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(145)) - Edits file /hadoop/hdfs/namesecondary/current/edits_0000000000020154770-0000000000020154772 of size 121 edits # 3 loaded in 0 seconds
2018-04-25 19:03:47,647 INFO namenode.FSImageFormatProtobuf (FSImageFormatProtobuf.java:save(413)) - Saving image file /hadoop/hdfs/namesecondary/current/fsimage.ckpt_0000000000020154772 using no compression
2018-04-25 19:03:47,783 INFO namenode.FSImageFormatProtobuf (FSImageFormatProtobuf.java:save(416)) - Image file /hadoop/hdfs/namesecondary/current/fsimage.ckpt_0000000000020154772 of size 336330 bytes saved in 0 seconds.
2018-04-25 19:03:47,818 INFO namenode.NNStorageRetentionManager (NNStorageRetentionManager.java:getImageTxIdToRetain(203)) - Going to retain 2 images with txid >= 20154766
2018-04-25 19:03:47,818 INFO namenode.NNStorageRetentionManager (NNStorageRetentionManager.java:purgeImage(225)) - Purging old image FSImageFile(file=/hadoop/hdfs/namesecondary/current/fsimage_0000000000020154762, cpktTxId=0000000000020154762)
2018-04-25 19:03:47,894 INFO namenode.TransferFsImage (TransferFsImage.java:copyFileToStream(395)) - Sending fileName: /hadoop/hdfs/namesecondary/current/fsimage_0000000000020154772, fileSize: 336330. Sent total: 336330 bytes. Size of last segment intended to send: -1 bytes.
2018-04-25 19:03:48,401 INFO namenode.TransferFsImage (TransferFsImage.java:uploadImageFromStorage(238)) - Uploaded image with txid 20154772 to namenode at http://master.root.com.mx:50070 in 0.522 seconds
2018-04-25 19:03:48,402 WARN namenode.SecondaryNameNode (SecondaryNameNode.java:doCheckpoint(576)) - Checkpoint done. New Image Size: 336330
==> /var/log/hadoop/hdfs/gc.log-201804251749 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(8717440k free), swap 4194300k(4141584k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T17:49:34.966-0500: 1.737: [GC (Allocation Failure) 2018-04-25T17:49:34.966-0500: 1.737: [ParNew: 163840K->12700K(184320K), 0.0122268 secs] 163840K->12700K(1028096K), 0.0123308 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
Heap
par new generation total 184320K, used 148567K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 82% used [0x00000000c0000000, 0x00000000c84aec40, 0x00000000ca000000)
from space 20480K, 62% used [0x00000000cb400000, 0x00000000cc067230, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25016K, capacity 25316K, committed 25612K, reserved 1071104K
class space used 2982K, capacity 3103K, committed 3144K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804051700 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(12812344k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-05T17:00:39.957-0500: 1.443: [GC (Allocation Failure) 2018-04-05T17:00:39.957-0500: 1.443: [ParNew: 104960K->11127K(118016K), 0.0148468 secs] 104960K->11127K(1035520K), 0.0149559 secs] [Times: user=0.05 sys=0.02, real=0.02 secs]
2018-04-05T17:01:40.942-0500: 62.427: [GC (Allocation Failure) 2018-04-05T17:01:40.942-0500: 62.427: [ParNew: 116087K->13056K(118016K), 0.0489811 secs] 116087K->25507K(1035520K), 0.0490985 secs] [Times: user=0.24 sys=0.01, real=0.05 secs]
2018-04-05T17:01:40.991-0500: 62.477: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12451K(917504K)] 26093K(1035520K), 0.0085256 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-05T17:01:41.000-0500: 62.486: [CMS-concurrent-mark-start]
2018-04-05T17:01:41.007-0500: 62.493: [CMS-concurrent-mark: 0.007/0.007 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
2018-04-05T17:01:41.007-0500: 62.493: [CMS-concurrent-preclean-start]
2018-04-05T17:01:41.010-0500: 62.495: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-05T17:01:41.010-0500: 62.495: [GC (CMS Final Remark) [YG occupancy: 13641 K (118016 K)]2018-04-05T17:01:41.010-0500: 62.495: [Rescan (parallel) , 0.0076868 secs]2018-04-05T17:01:41.018-0500: 62.503: [weak refs processing, 0.0003540 secs]2018-04-05T17:01:41.018-0500: 62.503: [class unloading, 0.0029434 secs]2018-04-05T17:01:41.021-0500: 62.506: [scrub symbol table, 0.0040624 secs]2018-04-05T17:01:41.025-0500: 62.510: [scrub string table, 0.0004494 secs][1 CMS-remark: 12451K(917504K)] 26093K(1035520K), 0.0161366 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2018-04-05T17:01:41.026-0500: 62.512: [CMS-concurrent-sweep-start]
2018-04-05T17:01:41.028-0500: 62.514: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-05T17:01:41.028-0500: 62.514: [CMS-concurrent-reset-start]
2018-04-05T17:01:41.032-0500: 62.518: [CMS-concurrent-reset: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2018-04-05T17:44:54.654-0500: 2656.140: [GC (Allocation Failure) 2018-04-05T17:44:54.654-0500: 2656.140: [ParNew: 118016K->3605K(118016K), 0.0540953 secs] 130460K->22989K(1035520K), 0.0541760 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
Heap
par new generation total 118016K, used 5180K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 1% used [0x00000000c0000000, 0x00000000c0189a08, 0x00000000c6680000)
from space 13056K, 27% used [0x00000000c7340000, 0x00000000c76c56e0, 0x00000000c8000000)
to space 13056K, 0% used [0x00000000c6680000, 0x00000000c6680000, 0x00000000c7340000)
concurrent mark-sweep generation total 917504K, used 19384K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25148K, capacity 25424K, committed 25676K, reserved 1073152K
class space used 2731K, capacity 2842K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804021614 <==
2018-04-04T23:58:09.777-0500: 200619.906: [GC (Allocation Failure) 2018-04-04T23:58:09.777-0500: 200619.906: [ParNew: 165598K->1570K(184320K), 0.0124203 secs] 208310K->44282K(1028096K), 0.0126470 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-04T23:59:31.131-0500: 200701.260: [GC (Allocation Failure) 2018-04-04T23:59:31.131-0500: 200701.260: [ParNew: 165410K->1687K(184320K), 0.0126581 secs] 208122K->44465K(1028096K), 0.0128360 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
2018-04-05T00:00:56.037-0500: 200786.166: [GC (Allocation Failure) 2018-04-05T00:00:56.037-0500: 200786.166: [ParNew: 165527K->1788K(184320K), 0.0127537 secs] 208305K->44566K(1028096K), 0.0129314 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:02:16.763-0500: 200866.892: [GC (Allocation Failure) 2018-04-05T00:02:16.763-0500: 200866.892: [ParNew: 165628K->1781K(184320K), 0.0128139 secs] 208406K->44567K(1028096K), 0.0129871 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:03:42.943-0500: 200953.072: [GC (Allocation Failure) 2018-04-05T00:03:42.943-0500: 200953.072: [ParNew: 165621K->1827K(184320K), 0.0091156 secs] 208407K->44613K(1028096K), 0.0092963 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:05:02.998-0500: 201033.127: [GC (Allocation Failure) 2018-04-05T00:05:02.998-0500: 201033.127: [ParNew: 165667K->1851K(184320K), 0.0122283 secs] 208453K->44637K(1028096K), 0.0123983 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-05T00:06:29.542-0500: 201119.671: [GC (Allocation Failure) 2018-04-05T00:06:29.543-0500: 201119.672: [ParNew: 165691K->1610K(184320K), 0.0124777 secs] 208477K->44462K(1028096K), 0.0126540 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:07:47.138-0500: 201197.267: [GC (Allocation Failure) 2018-04-05T00:07:47.138-0500: 201197.267: [ParNew: 165450K->1527K(184320K), 0.0106502 secs] 208302K->44445K(1028096K), 0.0108284 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-05T00:09:11.110-0500: 201281.239: [GC (Allocation Failure) 2018-04-05T00:09:11.110-0500: 201281.239: [ParNew: 165367K->1653K(184320K), 0.0120266 secs] 208285K->44579K(1028096K), 0.0122033 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2018-04-05T00:10:32.635-0500: 201362.764: [GC (Allocation Failure) 2018-04-05T00:10:32.635-0500: 201362.764: [ParNew: 165493K->1695K(184320K), 0.0119206 secs] 208419K->44621K(1028096K), 0.0120879 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:11:54.811-0500: 201444.940: [GC (Allocation Failure) 2018-04-05T00:11:54.811-0500: 201444.940: [ParNew: 165535K->1563K(184320K), 0.0120542 secs] 208461K->44497K(1028096K), 0.0122483 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:13:17.515-0500: 201527.644: [GC (Allocation Failure) 2018-04-05T00:13:17.515-0500: 201527.644: [ParNew: 165403K->1659K(184320K), 0.0123245 secs] 208337K->44593K(1028096K), 0.0124961 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:14:40.195-0500: 201610.324: [GC (Allocation Failure) 2018-04-05T00:14:40.196-0500: 201610.324: [ParNew: 165499K->1713K(184320K), 0.0110904 secs] 208433K->44648K(1028096K), 0.0112703 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:16:02.560-0500: 201692.689: [GC (Allocation Failure) 2018-04-05T00:16:02.560-0500: 201692.689: [ParNew: 165553K->1667K(184320K), 0.0095648 secs] 208488K->44601K(1028096K), 0.0097244 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:17:23.132-0500: 201773.261: [GC (Allocation Failure) 2018-04-05T00:17:23.132-0500: 201773.261: [ParNew: 165507K->1605K(184320K), 0.0130072 secs] 208441K->44604K(1028096K), 0.0131798 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:19:11.062-0500: 201881.191: [GC (Allocation Failure) 2018-04-05T00:19:11.063-0500: 201881.191: [ParNew: 165445K->1704K(184320K), 0.0121952 secs] 208444K->44703K(1028096K), 0.0123901 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:20:31.226-0500: 201961.355: [GC (Allocation Failure) 2018-04-05T00:20:31.226-0500: 201961.355: [ParNew: 165544K->1724K(184320K), 0.0120857 secs] 208543K->44724K(1028096K), 0.0122559 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T00:28:51.554-0500: 202461.683: [GC (Allocation Failure) 2018-04-05T00:28:51.554-0500: 202461.683: [ParNew: 165564K->2150K(184320K), 0.0115732 secs] 208564K->45150K(1028096K), 0.0117457 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T01:24:21.554-0500: 205791.683: [GC (Allocation Failure) 2018-04-05T01:24:21.554-0500: 205791.683: [ParNew: 165990K->1000K(184320K), 0.0089878 secs] 208990K->44000K(1028096K), 0.0091605 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T02:21:04.898-0500: 209195.027: [GC (Allocation Failure) 2018-04-05T02:21:04.898-0500: 209195.027: [ParNew: 164840K->767K(184320K), 0.0065875 secs] 207840K->43767K(1028096K), 0.0067286 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-05T03:09:00.718-0500: 212070.847: [GC (Allocation Failure) 2018-04-05T03:09:00.718-0500: 212070.847: [ParNew: 164607K->1813K(184320K), 0.0063541 secs] 207607K->44814K(1028096K), 0.0065176 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-05T04:07:21.556-0500: 215571.685: [GC (Allocation Failure) 2018-04-05T04:07:21.556-0500: 215571.685: [ParNew: 165653K->1435K(184320K), 0.0082442 secs] 208654K->44436K(1028096K), 0.0084204 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T05:06:29.844-0500: 219119.973: [GC (Allocation Failure) 2018-04-05T05:06:29.844-0500: 219119.973: [ParNew: 165275K->919K(184320K), 0.0062758 secs] 208276K->43920K(1028096K), 0.0064222 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T06:04:52.396-0500: 222622.525: [GC (Allocation Failure) 2018-04-05T06:04:52.397-0500: 222622.525: [ParNew: 164759K->947K(184320K), 0.0093398 secs] 207760K->44122K(1028096K), 0.0095633 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T07:02:32.472-0500: 226082.601: [GC (Allocation Failure) 2018-04-05T07:02:32.472-0500: 226082.601: [ParNew: 164787K->692K(184320K), 0.0045548 secs] 207962K->43944K(1028096K), 0.0046755 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T08:01:57.231-0500: 229647.360: [GC (Allocation Failure) 2018-04-05T08:01:57.232-0500: 229647.361: [ParNew: 164532K->753K(184320K), 0.0080061 secs] 207784K->44007K(1028096K), 0.0081526 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-05T09:00:33.735-0500: 233163.864: [GC (Allocation Failure) 2018-04-05T09:00:33.735-0500: 233163.864: [ParNew: 164593K->769K(184320K), 0.0077805 secs] 207847K->44024K(1028096K), 0.0079760 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T09:48:42.015-0500: 236052.144: [GC (Allocation Failure) 2018-04-05T09:48:42.015-0500: 236052.144: [ParNew: 164609K->481K(184320K), 0.0054391 secs] 207864K->43898K(1028096K), 0.0055401 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T10:48:17.480-0500: 239627.609: [GC (Allocation Failure) 2018-04-05T10:48:17.480-0500: 239627.609: [ParNew: 164321K->595K(184320K), 0.0085697 secs] 207738K->44014K(1028096K), 0.0087455 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T11:43:54.805-0500: 242964.934: [GC (Allocation Failure) 2018-04-05T11:43:54.805-0500: 242964.934: [ParNew: 164435K->786K(184320K), 0.0082437 secs] 207854K->44207K(1028096K), 0.0084188 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-05T12:30:46.558-0500: 245776.687: [GC (Allocation Failure) 2018-04-05T12:30:46.558-0500: 245776.687: [ParNew: 164626K->933K(184320K), 0.0072075 secs] 208047K->44354K(1028096K), 0.0073827 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T13:17:42.043-0500: 248592.172: [GC (Allocation Failure) 2018-04-05T13:17:42.043-0500: 248592.172: [ParNew: 164773K->835K(184320K), 0.0047785 secs] 208194K->44256K(1028096K), 0.0048990 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
Heap
par new generation total 184320K, used 17192K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 9% used [0x00000000c0000000, 0x00000000c0ff9570, 0x00000000ca000000)
from space 20480K, 4% used [0x00000000cb400000, 0x00000000cb4d0ca0, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 43421K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 47276K, capacity 48058K, committed 48344K, reserved 1093632K
class space used 4886K, capacity 5053K, committed 5140K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.out.5 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804021300 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(3068680k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T13:00:47.133-0500: 1.475: [GC (Allocation Failure) 2018-04-02T13:00:47.133-0500: 1.476: [ParNew: 104960K->11115K(118016K), 0.0128225 secs] 104960K->11115K(1035520K), 0.0129248 secs] [Times: user=0.05 sys=0.01, real=0.01 secs]
2018-04-02T13:01:48.108-0500: 62.451: [GC (Allocation Failure) 2018-04-02T13:01:48.109-0500: 62.451: [ParNew: 116075K->13056K(118016K), 0.0744387 secs] 116075K->25481K(1035520K), 0.0745758 secs] [Times: user=0.24 sys=0.02, real=0.08 secs]
2018-04-02T13:01:48.184-0500: 62.527: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12425K(917504K)] 26060K(1035520K), 0.0117795 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-02T13:01:48.196-0500: 62.539: [CMS-concurrent-mark-start]
2018-04-02T13:01:48.209-0500: 62.552: [CMS-concurrent-mark: 0.013/0.013 secs] [Times: user=0.07 sys=0.01, real=0.01 secs]
2018-04-02T13:01:48.209-0500: 62.552: [CMS-concurrent-preclean-start]
2018-04-02T13:01:48.211-0500: 62.554: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-02T13:01:48.211-0500: 62.554: [GC (CMS Final Remark) [YG occupancy: 13635 K (118016 K)]2018-04-02T13:01:48.211-0500: 62.554: [Rescan (parallel) , 0.0080054 secs]2018-04-02T13:01:48.219-0500: 62.562: [weak refs processing, 0.0000492 secs]2018-04-02T13:01:48.219-0500: 62.562: [class unloading, 0.0032907 secs]2018-04-02T13:01:48.223-0500: 62.566: [scrub symbol table, 0.0043958 secs]2018-04-02T13:01:48.227-0500: 62.570: [scrub string table, 0.0004947 secs][1 CMS-remark: 12425K(917504K)] 26060K(1035520K), 0.0169974 secs] [Times: user=0.07 sys=0.00, real=0.01 secs]
2018-04-02T13:01:48.228-0500: 62.571: [CMS-concurrent-sweep-start]
2018-04-02T13:01:48.231-0500: 62.574: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-02T13:01:48.231-0500: 62.574: [CMS-concurrent-reset-start]
2018-04-02T13:01:48.241-0500: 62.584: [CMS-concurrent-reset: 0.010/0.010 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
Heap
par new generation total 118016K, used 75409K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 59% used [0x00000000c0000000, 0x00000000c3ce4548, 0x00000000c6680000)
from space 13056K, 100% used [0x00000000c6680000, 0x00000000c7340000, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 12420K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 24961K, capacity 25232K, committed 25676K, reserved 1073152K
class space used 2739K, capacity 2842K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804051759 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(13589024k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-05T17:59:52.605-0500: 1.802: [GC (Allocation Failure) 2018-04-05T17:59:52.605-0500: 1.802: [ParNew: 104960K->11100K(118016K), 0.0167637 secs] 104960K->11100K(1035520K), 0.0168832 secs] [Times: user=0.06 sys=0.01, real=0.02 secs]
2018-04-05T18:00:54.060-0500: 63.257: [GC (Allocation Failure) 2018-04-05T18:00:54.060-0500: 63.257: [ParNew: 116060K->13056K(118016K), 0.0664771 secs] 116060K->25512K(1035520K), 0.0665811 secs] [Times: user=0.32 sys=0.01, real=0.06 secs]
2018-04-05T18:00:54.127-0500: 63.324: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12456K(917504K)] 26098K(1035520K), 0.0095432 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-05T18:00:54.137-0500: 63.334: [CMS-concurrent-mark-start]
2018-04-05T18:00:54.146-0500: 63.343: [CMS-concurrent-mark: 0.009/0.009 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
2018-04-05T18:00:54.146-0500: 63.343: [CMS-concurrent-preclean-start]
2018-04-05T18:00:54.150-0500: 63.347: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-05T18:00:54.150-0500: 63.347: [GC (CMS Final Remark) [YG occupancy: 13641 K (118016 K)]2018-04-05T18:00:54.150-0500: 63.347: [Rescan (parallel) , 0.0072177 secs]2018-04-05T18:00:54.157-0500: 63.354: [weak refs processing, 0.0004474 secs]2018-04-05T18:00:54.158-0500: 63.355: [class unloading, 0.0037489 secs]2018-04-05T18:00:54.161-0500: 63.358: [scrub symbol table, 0.0051956 secs]2018-04-05T18:00:54.167-0500: 63.364: [scrub string table, 0.0005854 secs][1 CMS-remark: 12456K(917504K)] 26098K(1035520K), 0.0181436 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
2018-04-05T18:00:54.168-0500: 63.365: [CMS-concurrent-sweep-start]
2018-04-05T18:00:54.172-0500: 63.369: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-05T18:00:54.172-0500: 63.369: [CMS-concurrent-reset-start]
2018-04-05T18:00:54.177-0500: 63.374: [CMS-concurrent-reset: 0.006/0.006 secs] [Times: user=0.01 sys=0.01, real=0.00 secs]
2018-04-05T18:50:55.313-0500: 3064.510: [GC (Allocation Failure) 2018-04-05T18:50:55.313-0500: 3064.510: [ParNew: 118016K->3857K(118016K), 0.0511732 secs] 130465K->23248K(1035520K), 0.0513094 secs] [Times: user=0.38 sys=0.00, real=0.05 secs]
2018-04-05T21:05:55.833-0500: 11165.030: [GC (Allocation Failure) 2018-04-05T21:05:55.833-0500: 11165.030: [ParNew: 108817K->1591K(118016K), 0.0058144 secs] 128208K->20982K(1035520K), 0.0059370 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T23:41:56.300-0500: 20525.497: [GC (Allocation Failure) 2018-04-05T23:41:56.300-0500: 20525.497: [ParNew: 106551K->1764K(118016K), 0.0063617 secs] 125942K->21155K(1035520K), 0.0064989 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]
2018-04-06T02:29:57.227-0500: 30606.424: [GC (Allocation Failure) 2018-04-06T02:29:57.227-0500: 30606.425: [ParNew: 106724K->1992K(118016K), 0.0060863 secs] 126115K->21383K(1035520K), 0.0062256 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T05:24:57.696-0500: 41106.894: [GC (Allocation Failure) 2018-04-06T05:24:57.696-0500: 41106.894: [ParNew: 106952K->1910K(118016K), 0.0049129 secs] 126343K->21301K(1035520K), 0.0050238 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T08:32:22.616-0500: 52351.813: [GC (Allocation Failure) 2018-04-06T08:32:22.616-0500: 52351.813: [ParNew: 106870K->1950K(118016K), 0.0074080 secs] 126261K->21341K(1035520K), 0.0075913 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-06T11:51:59.102-0500: 64328.299: [GC (Allocation Failure) 2018-04-06T11:51:59.102-0500: 64328.299: [ParNew: 106910K->1458K(118016K), 0.0100405 secs] 126301K->21818K(1035520K), 0.0101856 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-06T14:59:00.011-0500: 75549.208: [GC (Allocation Failure) 2018-04-06T14:59:00.011-0500: 75549.208: [ParNew: 106418K->708K(118016K), 0.0086373 secs] 126778K->21073K(1035520K), 0.0087766 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
Heap
par new generation total 118016K, used 46302K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 43% used [0x00000000c0000000, 0x00000000c2c869f8, 0x00000000c6680000)
from space 13056K, 5% used [0x00000000c6680000, 0x00000000c6731088, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 20365K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 26254K, capacity 26576K, committed 26956K, reserved 1073152K
class space used 2737K, capacity 2842K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804021321 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(2279496k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T13:21:09.338-0500: 1.852: [GC (Allocation Failure) 2018-04-02T13:21:09.338-0500: 1.853: [ParNew: 163840K->13899K(184320K), 0.0220986 secs] 163840K->13899K(1028096K), 0.0222458 secs] [Times: user=0.07 sys=0.01, real=0.02 secs]
2018-04-02T13:21:11.360-0500: 3.874: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 118018K(1028096K), 0.0154261 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-02T13:21:11.375-0500: 3.890: [CMS-concurrent-mark-start]
2018-04-02T13:21:11.385-0500: 3.899: [CMS-concurrent-mark: 0.009/0.009 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2018-04-02T13:21:11.385-0500: 3.899: [CMS-concurrent-preclean-start]
2018-04-02T13:21:11.388-0500: 3.903: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T13:21:11.388-0500: 3.903: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2018-04-02T13:21:16.426-0500: 8.941: [CMS-concurrent-abortable-preclean: 2.207/5.038 secs] [Times: user=4.48 sys=0.09, real=5.04 secs]
2018-04-02T13:21:16.426-0500: 8.941: [GC (CMS Final Remark) [YG occupancy: 177739 K (184320 K)]2018-04-02T13:21:16.426-0500: 8.941: [Rescan (parallel) , 0.0172699 secs]2018-04-02T13:21:16.444-0500: 8.958: [weak refs processing, 0.0000316 secs]2018-04-02T13:21:16.444-0500: 8.958: [class unloading, 0.0045643 secs]2018-04-02T13:21:16.448-0500: 8.963: [scrub symbol table, 0.0050201 secs]2018-04-02T13:21:16.453-0500: 8.968: [scrub string table, 0.0003734 secs][1 CMS-remark: 0K(843776K)] 177739K(1028096K), 0.0280501 secs] [Times: user=0.08 sys=0.00, real=0.03 secs]
2018-04-02T13:21:16.455-0500: 8.969: [CMS-concurrent-sweep-start]
2018-04-02T13:21:16.455-0500: 8.969: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T13:21:16.455-0500: 8.969: [CMS-concurrent-reset-start]
2018-04-02T13:21:16.462-0500: 8.977: [CMS-concurrent-reset: 0.007/0.007 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2018-04-02T13:21:19.209-0500: 11.724: [GC (Allocation Failure) 2018-04-02T13:21:19.209-0500: 11.724: [ParNew: 177739K->14863K(184320K), 0.0651780 secs] 177739K->19259K(1028096K), 0.0653379 secs] [Times: user=0.19 sys=0.02, real=0.07 secs]
2018-04-02T13:36:10.045-0500: 902.559: [GC (Allocation Failure) 2018-04-02T13:36:10.045-0500: 902.559: [ParNew: 178703K->9567K(184320K), 0.0182980 secs] 183099K->19317K(1028096K), 0.0184593 secs] [Times: user=0.05 sys=0.01, real=0.02 secs]
Heap
par new generation total 184320K, used 29865K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 12% used [0x00000000c0000000, 0x00000000c13d2a00, 0x00000000ca000000)
from space 20480K, 46% used [0x00000000cb400000, 0x00000000cbd57cd0, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 9750K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 35245K, capacity 35658K, committed 35996K, reserved 1081344K
class space used 3939K, capacity 4065K, committed 4084K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804021137 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(6777516k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T11:37:06.732-0500: 1.449: [GC (Allocation Failure) 2018-04-02T11:37:06.732-0500: 1.449: [ParNew: 104960K->11079K(118016K), 0.0146152 secs] 104960K->11079K(1035520K), 0.0147263 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
2018-04-02T11:38:07.888-0500: 62.605: [GC (Allocation Failure) 2018-04-02T11:38:07.888-0500: 62.606: [ParNew: 116039K->13056K(118016K), 0.0684079 secs] 116039K->25509K(1035520K), 0.0684962 secs] [Times: user=0.44 sys=0.03, real=0.07 secs]
2018-04-02T11:38:07.957-0500: 62.674: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12453K(917504K)] 26094K(1035520K), 0.0073648 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-02T11:38:07.964-0500: 62.682: [CMS-concurrent-mark-start]
2018-04-02T11:38:07.973-0500: 62.690: [CMS-concurrent-mark: 0.009/0.009 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
2018-04-02T11:38:07.973-0500: 62.690: [CMS-concurrent-preclean-start]
2018-04-02T11:38:07.978-0500: 62.695: [CMS-concurrent-preclean: 0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T11:38:07.978-0500: 62.696: [GC (CMS Final Remark) [YG occupancy: 13641 K (118016 K)]2018-04-02T11:38:07.978-0500: 62.696: [Rescan (parallel) , 0.0075666 secs]2018-04-02T11:38:07.986-0500: 62.703: [weak refs processing, 0.0000688 secs]2018-04-02T11:38:07.986-0500: 62.703: [class unloading, 0.0031449 secs]2018-04-02T11:38:07.989-0500: 62.706: [scrub symbol table, 0.0043603 secs]2018-04-02T11:38:07.993-0500: 62.711: [scrub string table, 0.0005075 secs][1 CMS-remark: 12453K(917504K)] 26094K(1035520K), 0.0164184 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-02T11:38:07.995-0500: 62.712: [CMS-concurrent-sweep-start]
2018-04-02T11:38:07.998-0500: 62.715: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T11:38:07.998-0500: 62.715: [CMS-concurrent-reset-start]
2018-04-02T11:38:08.007-0500: 62.724: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2018-04-02T12:46:31.478-0500: 4166.196: [GC (Allocation Failure) 2018-04-02T12:46:31.478-0500: 4166.196: [ParNew: 118016K->3199K(118016K), 0.0607361 secs] 130462K->22584K(1035520K), 0.0608390 secs] [Times: user=0.39 sys=0.00, real=0.06 secs]
Heap
par new generation total 118016K, used 20066K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 16% used [0x00000000c0000000, 0x00000000c1078d18, 0x00000000c6680000)
from space 13056K, 24% used [0x00000000c7340000, 0x00000000c765fca0, 0x00000000c8000000)
to space 13056K, 0% used [0x00000000c6680000, 0x00000000c6680000, 0x00000000c7340000)
concurrent mark-sweep generation total 917504K, used 19384K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 24877K, capacity 25176K, committed 25420K, reserved 1071104K
class space used 2730K, capacity 2842K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804021621 <==
2018-04-04T23:36:56.645-0500: 198935.303: [GC (Allocation Failure) 2018-04-04T23:36:56.645-0500: 198935.303: [ParNew: 104988K->29K(118016K), 0.0033023 secs] 127345K->22388K(1035520K), 0.0034021 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-04T23:36:56.819-0500: 198935.477: [GC (Allocation Failure) 2018-04-04T23:36:56.819-0500: 198935.477: [ParNew: 104989K->35K(118016K), 0.0031131 secs] 127348K->22396K(1035520K), 0.0032460 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-04T23:36:56.995-0500: 198935.653: [GC (Allocation Failure) 2018-04-04T23:36:56.995-0500: 198935.653: [ParNew: 104995K->29K(118016K), 0.0038065 secs] 127356K->22393K(1035520K), 0.0039169 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-04T23:36:57.169-0500: 198935.827: [GC (Allocation Failure) 2018-04-04T23:36:57.169-0500: 198935.827: [ParNew: 104989K->29K(118016K), 0.0039238 secs] 127353K->22395K(1035520K), 0.0040310 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-04T23:36:57.346-0500: 198936.003: [GC (Allocation Failure) 2018-04-04T23:36:57.346-0500: 198936.003: [ParNew: 104989K->37K(118016K), 0.0030593 secs] 127355K->22405K(1035520K), 0.0031689 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-04T23:36:57.520-0500: 198936.178: [GC (Allocation Failure) 2018-04-04T23:36:57.520-0500: 198936.178: [ParNew: 104997K->33K(118016K), 0.0090343 secs] 127365K->22404K(1035520K), 0.0091706 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
2018-04-04T23:36:57.701-0500: 198936.359: [GC (Allocation Failure) 2018-04-04T23:36:57.701-0500: 198936.359: [ParNew: 104993K->35K(118016K), 0.0033960 secs] 127364K->22407K(1035520K), 0.0034982 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-04T23:36:57.878-0500: 198936.535: [GC (Allocation Failure) 2018-04-04T23:36:57.878-0500: 198936.535: [ParNew: 104995K->30K(118016K), 0.0038801 secs] 127367K->22405K(1035520K), 0.0040126 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-04T23:36:58.053-0500: 198936.711: [GC (Allocation Failure) 2018-04-04T23:36:58.053-0500: 198936.711: [ParNew: 104990K->33K(118016K), 0.0037583 secs] 127365K->22410K(1035520K), 0.0038673 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-04T23:36:58.229-0500: 198936.887: [GC (Allocation Failure) 2018-04-04T23:36:58.229-0500: 198936.887: [ParNew: 104993K->33K(118016K), 0.0075792 secs] 127370K->22413K(1035520K), 0.0076810 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-04T23:36:58.407-0500: 198937.065: [GC (Allocation Failure) 2018-04-04T23:36:58.407-0500: 198937.065: [ParNew: 104993K->33K(118016K), 0.0040968 secs] 127373K->22415K(1035520K), 0.0041971 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-04T23:36:58.582-0500: 198937.240: [GC (Allocation Failure) 2018-04-04T23:36:58.583-0500: 198937.240: [ParNew: 104993K->39K(118016K), 0.0035643 secs] 127375K->22423K(1035520K), 0.0036812 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-04T23:36:58.759-0500: 198937.417: [GC (Allocation Failure) 2018-04-04T23:36:58.759-0500: 198937.417: [ParNew: 104999K->30K(118016K), 0.0062380 secs] 127383K->22416K(1035520K), 0.0063449 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-04T23:36:58.938-0500: 198937.596: [GC (Allocation Failure) 2018-04-04T23:36:59.077-0500: 198937.734: [ParNew: 104990K->28K(118016K), 0.0037280 secs] 127376K->22417K(1035520K), 0.1422586 secs] [Times: user=0.03 sys=0.00, real=0.14 secs]
2018-04-04T23:36:59.254-0500: 198937.912: [GC (Allocation Failure) 2018-04-04T23:36:59.254-0500: 198937.912: [ParNew: 104988K->34K(118016K), 0.0041560 secs] 127377K->22426K(1035520K), 0.0042116 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-04T23:36:59.429-0500: 198938.087: [GC (Allocation Failure) 2018-04-04T23:36:59.429-0500: 198938.087: [ParNew: 104994K->31K(118016K), 0.0031273 secs] 127386K->22426K(1035520K), 0.0031892 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-04T23:36:59.716-0500: 198938.374: [GC (Allocation Failure) 2018-04-04T23:36:59.716-0500: 198938.374: [ParNew: 104990K->32K(118016K), 0.0030142 secs] 127384K->22429K(1035520K), 0.0031184 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-04T23:52:31.487-0500: 199870.144: [GC (Allocation Failure) 2018-04-04T23:52:31.487-0500: 199870.144: [ParNew: 104992K->83K(118016K), 0.0045086 secs] 127389K->22483K(1035520K), 0.0046198 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-05T00:40:31.393-0500: 202750.051: [GC (Allocation Failure) 2018-04-05T00:40:31.394-0500: 202750.051: [ParNew: 105043K->152K(118016K), 0.0045180 secs] 127443K->22555K(1035520K), 0.0046730 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-05T01:52:00.367-0500: 207039.025: [GC (Allocation Failure) 2018-04-05T01:52:00.367-0500: 207039.025: [ParNew: 105112K->228K(118016K), 0.0046984 secs] 127515K->22633K(1035520K), 0.0048507 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T03:04:00.524-0500: 211359.182: [GC (Allocation Failure) 2018-04-05T03:04:00.524-0500: 211359.182: [ParNew: 105188K->231K(118016K), 0.0042526 secs] 127593K->22638K(1035520K), 0.0043766 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T04:15:31.388-0500: 215650.046: [GC (Allocation Failure) 2018-04-05T04:15:31.388-0500: 215650.046: [ParNew: 105191K->210K(118016K), 0.0046756 secs] 127598K->22619K(1035520K), 0.0048076 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-05T05:37:01.304-0500: 220539.962: [GC (Allocation Failure) 2018-04-05T05:37:01.305-0500: 220539.962: [ParNew: 105170K->295K(118016K), 0.0043144 secs] 127579K->22710K(1035520K), 0.0044383 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T05:37:01.541-0500: 220540.199: [GC (Allocation Failure) 2018-04-05T05:37:01.541-0500: 220540.199: [ParNew: 105255K->121K(118016K), 0.0031252 secs] 127670K->22550K(1035520K), 0.0032253 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T05:37:01.711-0500: 220540.369: [GC (Allocation Failure) 2018-04-05T05:37:01.711-0500: 220540.369: [ParNew: 105081K->95K(118016K), 0.0033134 secs] 127510K->22525K(1035520K), 0.0034229 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T05:37:01.882-0500: 220540.540: [GC (Allocation Failure) 2018-04-05T05:37:01.883-0500: 220540.540: [ParNew: 105055K->98K(118016K), 0.0029602 secs] 127485K->22528K(1035520K), 0.0030734 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T05:37:02.052-0500: 220540.710: [GC (Allocation Failure) 2018-04-05T05:37:02.052-0500: 220540.710: [ParNew: 105058K->102K(118016K), 0.0030433 secs] 127488K->22532K(1035520K), 0.0031794 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-05T06:17:31.413-0500: 222970.071: [GC (Allocation Failure) 2018-04-05T06:17:31.413-0500: 222970.071: [ParNew: 105062K->149K(118016K), 0.0054817 secs] 127492K->22578K(1035520K), 0.0055930 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-05T07:29:27.450-0500: 227286.108: [GC (Allocation Failure) 2018-04-05T07:29:27.450-0500: 227286.108: [ParNew: 105109K->222K(118016K), 0.0046485 secs] 127538K->22653K(1035520K), 0.0047639 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T09:04:03.046-0500: 232961.704: [GC (Allocation Failure) 2018-04-05T09:04:03.046-0500: 232961.704: [ParNew: 105182K->273K(118016K), 0.0053261 secs] 127613K->22706K(1035520K), 0.0054550 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T10:38:03.436-0500: 238602.093: [GC (Allocation Failure) 2018-04-05T10:38:03.436-0500: 238602.093: [ParNew: 105233K->276K(118016K), 0.0047979 secs] 127666K->22711K(1035520K), 0.0049379 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-05T12:04:31.391-0500: 243790.049: [GC (Allocation Failure) 2018-04-05T12:04:31.391-0500: 243790.049: [ParNew: 105236K->251K(118016K), 0.0042367 secs] 127671K->22687K(1035520K), 0.0043149 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
Heap
par new generation total 118016K, used 90322K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 85% used [0x00000000c0000000, 0x00000000c57f5c58, 0x00000000c6680000)
from space 13056K, 1% used [0x00000000c6680000, 0x00000000c66beeb0, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 22436K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 26647K, capacity 26944K, committed 27212K, reserved 1073152K
class space used 2760K, capacity 2874K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804051754 <==
2018-04-05T17:54:41.382-0500: 9.315: [GC (CMS Final Remark) [YG occupancy: 143679 K (184320 K)]2018-04-05T17:54:41.382-0500: 9.315: [Rescan (parallel) , 0.0149394 secs]2018-04-05T17:54:41.397-0500: 9.330: [weak refs processing, 0.0000201 secs]2018-04-05T17:54:41.397-0500: 9.330: [class unloading, 0.0030261 secs]2018-04-05T17:54:41.400-0500: 9.333: [scrub symbol table, 0.0043732 secs]2018-04-05T17:54:41.404-0500: 9.337: [scrub string table, 0.0003126 secs][1 CMS-remark: 0K(843776K)] 143679K(1028096K), 0.0232984 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-05T17:54:41.405-0500: 9.339: [CMS-concurrent-sweep-start]
2018-04-05T17:54:41.405-0500: 9.339: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-05T17:54:41.405-0500: 9.339: [CMS-concurrent-reset-start]
2018-04-05T17:54:41.409-0500: 9.342: [CMS-concurrent-reset: 0.004/0.004 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2018-04-05T17:54:42.987-0500: 10.921: [GC (Allocation Failure) 2018-04-05T17:54:42.987-0500: 10.921: [ParNew: 176541K->15137K(184320K), 0.0416570 secs] 176541K->18911K(1028096K), 0.0417832 secs] [Times: user=0.13 sys=0.01, real=0.05 secs]
2018-04-05T18:11:56.568-0500: 1044.501: [GC (Allocation Failure) 2018-04-05T18:11:56.568-0500: 1044.501: [ParNew: 178977K->7199K(184320K), 0.0226691 secs] 182751K->17038K(1028096K), 0.0228289 secs] [Times: user=0.07 sys=0.00, real=0.03 secs]
2018-04-05T18:59:06.923-0500: 3874.857: [GC (Allocation Failure) 2018-04-05T18:59:06.923-0500: 3874.857: [ParNew: 171039K->3881K(184320K), 0.0104989 secs] 180878K->13720K(1028096K), 0.0106621 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T19:54:35.798-0500: 7203.731: [GC (Allocation Failure) 2018-04-05T19:54:35.798-0500: 7203.731: [ParNew: 167721K->2220K(184320K), 0.0090567 secs] 177560K->12059K(1028096K), 0.0092008 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T20:51:15.606-0500: 10603.539: [GC (Allocation Failure) 2018-04-05T20:51:15.606-0500: 10603.539: [ParNew: 166060K->2308K(184320K), 0.0109233 secs] 175899K->12146K(1028096K), 0.0110867 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-05T21:48:05.606-0500: 14013.540: [GC (Allocation Failure) 2018-04-05T21:48:05.606-0500: 14013.540: [ParNew: 166148K->2602K(184320K), 0.0088557 secs] 175986K->12441K(1028096K), 0.0090715 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
2018-04-05T22:44:25.609-0500: 17393.542: [GC (Allocation Failure) 2018-04-05T22:44:25.609-0500: 17393.543: [ParNew: 166442K->2901K(184320K), 0.0114017 secs] 176281K->12740K(1028096K), 0.0115648 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-05T23:29:45.606-0500: 20113.539: [GC (Allocation Failure) 2018-04-05T23:29:45.606-0500: 20113.539: [ParNew: 166741K->1869K(184320K), 0.0126643 secs] 176580K->12653K(1028096K), 0.0128366 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2018-04-06T00:27:25.608-0500: 23573.541: [GC (Allocation Failure) 2018-04-06T00:27:25.608-0500: 23573.541: [ParNew: 165709K->1048K(184320K), 0.0055222 secs] 176493K->12256K(1028096K), 0.0056378 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-06T01:25:35.607-0500: 27063.541: [GC (Allocation Failure) 2018-04-06T01:25:35.608-0500: 27063.541: [ParNew: 164888K->849K(184320K), 0.0089897 secs] 176096K->12075K(1028096K), 0.0091458 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T02:24:22.209-0500: 30590.143: [GC (Allocation Failure) 2018-04-06T02:24:22.209-0500: 30590.143: [ParNew: 164689K->789K(184320K), 0.0086972 secs] 175915K->12037K(1028096K), 0.0088944 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-06T03:22:22.525-0500: 34070.458: [GC (Allocation Failure) 2018-04-06T03:22:22.525-0500: 34070.458: [ParNew: 164629K->817K(184320K), 0.0091885 secs] 175877K->12079K(1028096K), 0.0094080 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T04:20:15.607-0500: 37543.541: [GC (Allocation Failure) 2018-04-06T04:20:15.608-0500: 37543.541: [ParNew: 164657K->744K(184320K), 0.0089939 secs] 175919K->12033K(1028096K), 0.0091595 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T05:18:45.607-0500: 41053.540: [GC (Allocation Failure) 2018-04-06T05:18:45.607-0500: 41053.540: [ParNew: 164584K->738K(184320K), 0.0095319 secs] 175873K->12036K(1028096K), 0.0096998 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T06:05:01.790-0500: 43829.724: [GC (Allocation Failure) 2018-04-06T06:05:01.790-0500: 43829.724: [ParNew: 164578K->586K(184320K), 0.0090991 secs] 175876K->12052K(1028096K), 0.0092852 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T07:02:45.646-0500: 47293.580: [GC (Allocation Failure) 2018-04-06T07:02:45.646-0500: 47293.580: [ParNew: 164426K->558K(184320K), 0.0092489 secs] 175892K->12031K(1028096K), 0.0094114 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T08:01:01.052-0500: 50788.986: [GC (Allocation Failure) 2018-04-06T08:01:01.052-0500: 50788.986: [ParNew: 164398K->510K(184320K), 0.0091999 secs] 175871K->11984K(1028096K), 0.0093706 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T08:59:38.257-0500: 54306.190: [GC (Allocation Failure) 2018-04-06T08:59:38.257-0500: 54306.190: [ParNew: 164350K->508K(184320K), 0.0092500 secs] 175824K->11985K(1028096K), 0.0094360 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-06T09:58:22.732-0500: 57830.665: [GC (Allocation Failure) 2018-04-06T09:58:22.732-0500: 57830.665: [ParNew: 164348K->565K(184320K), 0.0073763 secs] 175825K->12045K(1028096K), 0.0075071 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-06T10:56:29.559-0500: 61317.492: [GC (Allocation Failure) 2018-04-06T10:56:29.559-0500: 61317.492: [ParNew: 164405K->808K(184320K), 0.0090292 secs] 175885K->12290K(1028096K), 0.0091898 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-06T11:42:45.941-0500: 64093.874: [GC (Allocation Failure) 2018-04-06T11:42:45.941-0500: 64093.874: [ParNew: 164648K->554K(184320K), 0.0088030 secs] 176130K->12038K(1028096K), 0.0089592 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T12:41:07.165-0500: 67595.098: [GC (Allocation Failure) 2018-04-06T12:41:07.165-0500: 67595.098: [ParNew: 164394K->750K(184320K), 0.0091729 secs] 175878K->12235K(1028096K), 0.0093344 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T13:22:07.411-0500: 70055.345: [GC (Allocation Failure) 2018-04-06T13:22:07.411-0500: 70055.345: [ParNew: 164590K->2661K(184320K), 0.0120979 secs] 176075K->14148K(1028096K), 0.0122916 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-06T14:09:25.775-0500: 72893.709: [GC (Allocation Failure) 2018-04-06T14:09:25.775-0500: 72893.709: [ParNew: 166501K->1284K(184320K), 0.0045499 secs] 177988K->12790K(1028096K), 0.0046591 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-06T14:55:46.123-0500: 75674.056: [GC (Allocation Failure) 2018-04-06T14:55:46.123-0500: 75674.056: [ParNew: 165124K->1585K(184320K), 0.0090661 secs] 176630K->13092K(1028096K), 0.0092384 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-06T15:42:33.309-0500: 78481.242: [GC (Allocation Failure) 2018-04-06T15:42:33.309-0500: 78481.242: [ParNew: 165425K->1429K(184320K), 0.0076720 secs] 176932K->12938K(1028096K), 0.0078391 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-06T16:08:50.610-0500: 80058.543: [GC (Allocation Failure) 2018-04-06T16:08:50.610-0500: 80058.543: [ParNew: 165269K->6032K(184320K), 0.0122660 secs] 176778K->17550K(1028096K), 0.0124198 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
Heap
par new generation total 184320K, used 12542K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 3% used [0x00000000c0000000, 0x00000000c065ba28, 0x00000000ca000000)
from space 20480K, 29% used [0x00000000ca000000, 0x00000000ca5e4010, 0x00000000cb400000)
to space 20480K, 0% used [0x00000000cb400000, 0x00000000cb400000, 0x00000000cc800000)
concurrent mark-sweep generation total 843776K, used 11518K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 42180K, capacity 42692K, committed 43096K, reserved 1087488K
class space used 4505K, capacity 4682K, committed 4756K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804251740 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(8734820k free), swap 4194300k(4141584k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T17:40:43.186-0500: 2.011: [GC (Allocation Failure) 2018-04-25T17:40:43.186-0500: 2.011: [ParNew: 163840K->12701K(184320K), 0.0146256 secs] 163840K->12701K(1028096K), 0.0147627 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
Heap
par new generation total 184320K, used 148560K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 82% used [0x00000000c0000000, 0x00000000c84acf88, 0x00000000ca000000)
from space 20480K, 62% used [0x00000000cb400000, 0x00000000cc067458, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25040K, capacity 25348K, committed 25612K, reserved 1071104K
class space used 2985K, capacity 3135K, committed 3144K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-nodo1.root.com.mx.out.3 <==
ulimit -a for user hdfs
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-journalnode-nodo1.root.com.mx.out.2 <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
g signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804021110 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(6574248k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T11:10:31.813-0500: 1.566: [GC (Allocation Failure) 2018-04-02T11:10:31.813-0500: 1.566: [ParNew: 104960K->11118K(118016K), 0.0119121 secs] 104960K->11118K(1035520K), 0.0120034 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
2018-04-02T11:11:32.976-0500: 62.730: [GC (Allocation Failure) 2018-04-02T11:11:32.976-0500: 62.730: [ParNew: 116078K->13055K(118016K), 0.0888761 secs] 116078K->25508K(1035520K), 0.0889729 secs] [Times: user=0.19 sys=0.03, real=0.09 secs]
2018-04-02T11:11:33.065-0500: 62.819: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12452K(917504K)] 26094K(1035520K), 0.0069461 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-04-02T11:11:33.072-0500: 62.826: [CMS-concurrent-mark-start]
2018-04-02T11:11:33.082-0500: 62.836: [CMS-concurrent-mark: 0.010/0.010 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-02T11:11:33.082-0500: 62.836: [CMS-concurrent-preclean-start]
2018-04-02T11:11:33.085-0500: 62.838: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-02T11:11:33.086-0500: 62.840: [GC (CMS Final Remark) [YG occupancy: 13641 K (118016 K)]2018-04-02T11:11:33.086-0500: 62.840: [Rescan (parallel) , 0.0077354 secs]2018-04-02T11:11:33.094-0500: 62.847: [weak refs processing, 0.0000427 secs]2018-04-02T11:11:33.094-0500: 62.847: [class unloading, 0.0029812 secs]2018-04-02T11:11:33.097-0500: 62.850: [scrub symbol table, 0.0045402 secs]2018-04-02T11:11:33.101-0500: 62.855: [scrub string table, 0.0005474 secs][1 CMS-remark: 12452K(917504K)] 26094K(1035520K), 0.0166454 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
2018-04-02T11:11:33.103-0500: 62.857: [CMS-concurrent-sweep-start]
2018-04-02T11:11:33.106-0500: 62.859: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T11:11:33.106-0500: 62.859: [CMS-concurrent-reset-start]
2018-04-02T11:11:33.115-0500: 62.869: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
Heap
par new generation total 118016K, used 74396K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 58% used [0x00000000c0000000, 0x00000000c3be7278, 0x00000000c6680000)
from space 13056K, 99% used [0x00000000c6680000, 0x00000000c733fff8, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 12445K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 24689K, capacity 24992K, committed 25420K, reserved 1071104K
class space used 2727K, capacity 2842K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.2 <==
2018-04-16 03:04:28,609 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40926, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777041_3036242, duration: 1970470
2018-04-16 03:04:28,609 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777041_3036242, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:28,770 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777042_3036243 src: /10.1.5.21:40930 dest: /10.1.5.20:50010
2018-04-16 03:04:28,773 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40930, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777042_3036243, duration: 1987570
2018-04-16 03:04:28,774 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777042_3036243, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:28,936 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777043_3036244 src: /10.1.5.22:41914 dest: /10.1.5.20:50010
2018-04-16 03:04:28,939 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:41914, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777043_3036244, duration: 2140204
2018-04-16 03:04:28,939 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777043_3036244, type=LAST_IN_PIPELINE terminating
2018-04-16 03:04:29,298 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777044_3036245 src: /10.1.5.22:41916 dest: /10.1.5.20:50010
2018-04-16 03:04:29,301 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:41916, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777044_3036245, duration: 2252440
2018-04-16 03:04:29,302 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777044_3036245, type=LAST_IN_PIPELINE terminating
2018-04-16 03:04:29,463 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777045_3036246 src: /10.1.5.21:40946 dest: /10.1.5.20:50010
2018-04-16 03:04:29,467 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40946, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777045_3036246, duration: 2047805
2018-04-16 03:04:29,467 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777045_3036246, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:29,630 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777046_3036247 src: /10.1.5.21:40950 dest: /10.1.5.20:50010
2018-04-16 03:04:29,634 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40950, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777046_3036247, duration: 1984293
2018-04-16 03:04:29,634 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777046_3036247, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:29,794 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777047_3036248 src: /10.1.5.21:40958 dest: /10.1.5.20:50010
2018-04-16 03:04:29,799 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40958, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777047_3036248, duration: 2184991
2018-04-16 03:04:29,799 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777047_3036248, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:29,969 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777048_3036249 src: /10.1.5.22:41920 dest: /10.1.5.20:50010
2018-04-16 03:04:29,972 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:41920, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777048_3036249, duration: 2013265
2018-04-16 03:04:29,972 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777048_3036249, type=LAST_IN_PIPELINE terminating
2018-04-16 03:04:30,133 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777049_3036250 src: /10.1.5.21:40970 dest: /10.1.5.20:50010
2018-04-16 03:04:30,137 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40970, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777049_3036250, duration: 1969165
2018-04-16 03:04:30,137 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777049_3036250, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:30,300 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777050_3036251 src: /10.1.5.21:40974 dest: /10.1.5.20:50010
2018-04-16 03:04:30,304 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40974, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777050_3036251, duration: 1914010
2018-04-16 03:04:30,304 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777050_3036251, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:30,466 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777051_3036252 src: /10.1.5.22:41922 dest: /10.1.5.20:50010
2018-04-16 03:04:30,469 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:41922, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777051_3036252, duration: 2168678
2018-04-16 03:04:30,469 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777051_3036252, type=LAST_IN_PIPELINE terminating
2018-04-16 03:04:30,630 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777052_3036253 src: /10.1.5.21:40992 dest: /10.1.5.20:50010
2018-04-16 03:04:30,634 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40992, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777052_3036253, duration: 1873797
2018-04-16 03:04:30,634 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777052_3036253, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:30,853 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777053_3036254 src: /10.1.5.21:40996 dest: /10.1.5.20:50010
2018-04-16 03:04:30,857 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:40996, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777053_3036254, duration: 2080631
2018-04-16 03:04:30,857 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076777053_3036254, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-16 03:04:31,042 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076777054_3036255 src: /10.1.5.22:41928 dest: /10.1.5.20:50010
2018-04-16 03:04:31,045 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:41928, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076777054_3036255, duration: 1969918
==> /var/log/hadoop/hdfs/gc.log-201804251729 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(9257248k free), swap 4194300k(4141584k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T17:29:40.844-0500: 2.946: [GC (Allocation Failure) 2018-04-25T17:29:40.844-0500: 2.946: [ParNew: 163840K->12708K(184320K), 0.0213360 secs] 163840K->12708K(1028096K), 0.0215185 secs] [Times: user=0.04 sys=0.01, real=0.03 secs]
2018-04-25T17:29:42.866-0500: 4.968: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 85962K(1028096K), 0.0136624 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-25T17:29:42.880-0500: 4.982: [CMS-concurrent-mark-start]
2018-04-25T17:29:42.889-0500: 4.991: [CMS-concurrent-mark: 0.009/0.009 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T17:29:42.889-0500: 4.991: [CMS-concurrent-preclean-start]
2018-04-25T17:29:42.892-0500: 4.994: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-04-25T17:29:42.892-0500: 4.994: [CMS-concurrent-abortable-preclean-start]
Heap
par new generation total 184320K, used 150346K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 84% used [0x00000000c0000000, 0x00000000c8669838, 0x00000000ca000000)
from space 20480K, 62% used [0x00000000cb400000, 0x00000000cc069380, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25073K, capacity 25418K, committed 25868K, reserved 1073152K
class space used 2992K, capacity 3136K, committed 3144K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log <==
2018-04-25 19:06:04,985 INFO http.HttpServer2 (HttpServer2.java:addGlobalFilter(791)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2018-04-25 19:06:04,987 INFO http.HttpServer2 (HttpServer2.java:addFilter(766)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
2018-04-25 19:06:04,987 INFO http.HttpServer2 (HttpServer2.java:addFilter(774)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2018-04-25 19:06:04,988 INFO http.HttpServer2 (HttpServer2.java:addFilter(774)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2018-04-25 19:06:04,988 INFO security.HttpCrossOriginFilterInitializer (HttpCrossOriginFilterInitializer.java:initFilter(49)) - CORS filter not enabled. Please set hadoop.http.cross-origin.enabled to 'true' to enable it
2018-04-25 19:06:04,999 INFO http.HttpServer2 (HttpServer2.java:bindListener(989)) - Jetty bound to port 42219
2018-04-25 19:06:04,999 INFO mortbay.log (Slf4jLog.java:info(67)) - jetty-6.1.26.hwx
2018-04-25 19:06:05,141 INFO mortbay.log (Slf4jLog.java:info(67)) - Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42219
2018-04-25 19:06:05,243 INFO web.DatanodeHttpServer (DatanodeHttpServer.java:start(233)) - Listening HTTP traffic on /0.0.0.0:50075
2018-04-25 19:06:05,245 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(179)) - Starting JVM pause monitor
2018-04-25 19:06:05,278 INFO datanode.DataNode (DataNode.java:startDataNode(1277)) - dnUserName = hdfs
2018-04-25 19:06:05,278 INFO datanode.DataNode (DataNode.java:startDataNode(1278)) - supergroup = hdfs
2018-04-25 19:06:05,343 INFO ipc.CallQueueManager (CallQueueManager.java:<init>(75)) - Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2018-04-25 19:06:05,355 INFO ipc.Server (Server.java:run(821)) - Starting Socket Reader #1 for port 8010
2018-04-25 19:06:05,405 INFO datanode.DataNode (DataNode.java:initIpcServer(941)) - Opened IPC server at /0.0.0.0:8010
2018-04-25 19:06:05,417 INFO datanode.DataNode (BlockPoolManager.java:refreshNamenodes(152)) - Refresh request received for nameservices: null
2018-04-25 19:06:05,436 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2018-04-25 19:06:05,537 INFO ipc.Server (Server.java:stop(2752)) - Stopping server on 8010
2018-04-25 19:06:05,538 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping DataNode metrics system...
2018-04-25 19:06:05,539 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.
2018-04-25 19:06:05,539 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - DataNode metrics system stopped.
2018-04-25 19:06:05,540 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - DataNode metrics system shutdown complete.
2018-04-25 19:06:05,542 INFO timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:getCurrentCollectorHost(230)) - No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.
2018-04-25 19:06:05,543 INFO datanode.DataNode (DataNode.java:shutdown(1987)) - Shutdown complete.
2018-04-25 19:06:05,543 ERROR datanode.DataNode (DataNode.java:secureMain(2692)) - Exception in secureMain
java.io.IOException: Unknown nameservice: namenodeha
at org.apache.hadoop.hdfs.DFSUtil.getNNServiceRpcAddressesForCluster(DFSUtil.java:879)
at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.refreshNamenodes(BlockPoolManager.java:156)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1287)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:456)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2591)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2493)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2540)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2685)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2709)
2018-04-25 19:06:05,544 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2018-04-25 19:06:05,546 INFO datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at nodo1.root.com.mx/10.1.5.20
************************************************************/
==> /var/log/hadoop/hdfs/gc.log-201804251806 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(8751604k free), swap 4194300k(4141584k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=209715200 -XX:MaxTenuringThreshold=6 -XX:NewSize=209715200 -XX:OldPLABSize=16 -XX:ParallelGCThreads=4 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-25T18:06:35.865-0500: 1.744: [GC (Allocation Failure) 2018-04-25T18:06:35.866-0500: 1.744: [ParNew: 163840K->12702K(184320K), 0.0144856 secs] 163840K->12702K(1028096K), 0.0145980 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-25T18:06:37.880-0500: 3.758: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(843776K)] 145295K(1028096K), 0.0123735 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-04-25T18:06:37.893-0500: 3.771: [CMS-concurrent-mark-start]
2018-04-25T18:06:37.899-0500: 3.777: [CMS-concurrent-mark: 0.006/0.006 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-25T18:06:37.900-0500: 3.778: [CMS-concurrent-preclean-start]
Heap
par new generation total 184320K, used 148572K [0x00000000c0000000, 0x00000000cc800000, 0x00000000cc800000)
eden space 163840K, 82% used [0x00000000c0000000, 0x00000000c84af6d8, 0x00000000ca000000)
from space 20480K, 62% used [0x00000000cb400000, 0x00000000cc067a80, 0x00000000cc800000)
to space 20480K, 0% used [0x00000000ca000000, 0x00000000ca000000, 0x00000000cb400000)
concurrent mark-sweep generation total 843776K, used 0K [0x00000000cc800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25046K, capacity 25412K, committed 25868K, reserved 1073152K
class space used 2986K, capacity 3135K, committed 3144K, reserved 1048576K
2018-04-25T18:06:37.903-0500: 3.781: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-nodo1.root.com.mx.out.1 <==
ulimit -a for user hdfs
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-nodo1.root.com.mx.log.3 <==
2018-04-15 16:17:39,327 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577250_2836451 src: /10.1.5.22:52284 dest: /10.1.5.20:50010
2018-04-15 16:17:39,330 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:52284, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577250_2836451, duration: 1976139
2018-04-15 16:17:39,330 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577250_2836451, type=LAST_IN_PIPELINE terminating
2018-04-15 16:17:39,492 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577251_2836452 src: /10.1.5.22:52288 dest: /10.1.5.20:50010
2018-04-15 16:17:39,495 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:52288, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577251_2836452, duration: 1892412
2018-04-15 16:17:39,495 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577251_2836452, type=LAST_IN_PIPELINE terminating
2018-04-15 16:17:39,656 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577252_2836453 src: /10.1.5.21:37746 dest: /10.1.5.20:50010
2018-04-15 16:17:39,660 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:37746, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577252_2836453, duration: 2069259
2018-04-15 16:17:39,660 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577252_2836453, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 16:17:39,902 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577253_2836454 src: /10.1.5.22:52290 dest: /10.1.5.20:50010
2018-04-15 16:17:39,905 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:52290, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577253_2836454, duration: 1972554
2018-04-15 16:17:39,905 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577253_2836454, type=LAST_IN_PIPELINE terminating
2018-04-15 16:17:40,078 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577254_2836455 src: /10.1.5.21:37756 dest: /10.1.5.20:50010
2018-04-15 16:17:40,082 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:37756, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577254_2836455, duration: 2240099
2018-04-15 16:17:40,082 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577254_2836455, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 16:17:40,236 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577255_2836456 src: /10.1.5.22:52292 dest: /10.1.5.20:50010
2018-04-15 16:17:40,271 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:52292, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577255_2836456, duration: 1918446
2018-04-15 16:17:40,271 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577255_2836456, type=LAST_IN_PIPELINE terminating
2018-04-15 16:17:40,444 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577256_2836457 src: /10.1.5.21:37764 dest: /10.1.5.20:50010
2018-04-15 16:17:40,447 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:37764, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577256_2836457, duration: 1760325
2018-04-15 16:17:40,447 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577256_2836457, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 16:17:41,041 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577257_2836458 src: /10.1.5.21:37772 dest: /10.1.5.20:50010
2018-04-15 16:17:41,044 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:37772, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577257_2836458, duration: 1437229
2018-04-15 16:17:41,044 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577257_2836458, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 16:17:41,198 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577258_2836459 src: /10.1.5.21:37776 dest: /10.1.5.20:50010
2018-04-15 16:17:41,201 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:37776, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577258_2836459, duration: 1557976
2018-04-15 16:17:41,201 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577258_2836459, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 16:17:42,207 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577259_2836460 src: /10.1.5.22:52308 dest: /10.1.5.20:50010
2018-04-15 16:17:42,210 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:52308, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577259_2836460, duration: 1757972
2018-04-15 16:17:42,210 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577259_2836460, type=LAST_IN_PIPELINE terminating
2018-04-15 16:17:42,372 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577260_2836461 src: /10.1.5.22:52310 dest: /10.1.5.20:50010
2018-04-15 16:17:42,375 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.22:52310, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577260_2836461, duration: 1651967
2018-04-15 16:17:42,375 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577260_2836461, type=LAST_IN_PIPELINE terminating
2018-04-15 16:17:42,743 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577261_2836462 src: /10.1.5.21:37798 dest: /10.1.5.20:50010
2018-04-15 16:17:42,746 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:37798, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577261_2836462, duration: 1263137
2018-04-15 16:17:42,746 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577261_2836462, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 16:17:42,908 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577262_2836463 src: /10.1.5.21:37802 dest: /10.1.5.20:50010
2018-04-15 16:17:42,911 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /10.1.5.21:37802, dest: /10.1.5.20:50010, bytes: 122, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-452404583_18, offset: 0, srvID: ef44e0ab-28af-4fa2-828e-2da7d9264cfc, blockid: BP-645840651-10.1.5.19-1522680031790:blk_1076577262_2836463, duration: 1398659
2018-04-15 16:17:42,911 INFO datanode.DataNode (BlockReceiver.java:run(1463)) - PacketResponder: BP-645840651-10.1.5.19-1522680031790:blk_1076577262_2836463, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[10.1.5.22:50010] terminating
2018-04-15 16:17:43,075 INFO datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-645840651-10.1.5.19-1522680031790:blk_1076577263_2836464 src: /10.1.5.22:52312 dest: /10.1.5.20:50010
==> /var/log/hadoop/hdfs/gc.log-201804101003 <==
2018-04-10T18:16:00.449-0500: 29525.930: [GC (Allocation Failure) 2018-04-10T18:16:00.449-0500: 29525.930: [ParNew: 106951K->2216K(118016K), 0.0075699 secs] 126465K->21730K(1035520K), 0.0077066 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]
2018-04-10T21:11:00.947-0500: 40026.428: [GC (Allocation Failure) 2018-04-10T21:11:00.947-0500: 40026.428: [ParNew: 107176K->1627K(118016K), 0.0052636 secs] 126690K->21141K(1035520K), 0.0053516 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-11T00:10:22.587-0500: 50788.068: [GC (Allocation Failure) 2018-04-11T00:10:22.587-0500: 50788.068: [ParNew: 106587K->2119K(118016K), 0.0077744 secs] 126101K->21633K(1035520K), 0.0079329 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-11T03:30:02.317-0500: 62767.798: [GC (Allocation Failure) 2018-04-11T03:30:02.317-0500: 62767.798: [ParNew: 107079K->1486K(118016K), 0.0096079 secs] 126593K->22004K(1035520K), 0.0097504 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
2018-04-11T06:31:22.555-0500: 73648.036: [GC (Allocation Failure) 2018-04-11T06:31:22.555-0500: 73648.036: [ParNew: 106446K->663K(118016K), 0.0073611 secs] 126964K->21189K(1035520K), 0.0075091 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-11T09:31:03.477-0500: 84428.958: [GC (Allocation Failure) 2018-04-11T09:31:03.477-0500: 84428.958: [ParNew: 105623K->553K(118016K), 0.0058886 secs] 126149K->21085K(1035520K), 0.0060099 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-11T11:34:04.106-0500: 91809.587: [GC (Allocation Failure) 2018-04-11T11:34:04.106-0500: 91809.587: [ParNew: 105513K->409K(118016K), 0.0054039 secs] 126045K->20964K(1035520K), 0.0055425 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-11T13:52:22.577-0500: 100108.058: [GC (Allocation Failure) 2018-04-11T13:52:22.577-0500: 100108.058: [ParNew: 105369K->382K(118016K), 0.0078189 secs] 125924K->20941K(1035520K), 0.0079520 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-11T16:05:05.004-0500: 108070.485: [GC (Allocation Failure) 2018-04-11T16:05:05.004-0500: 108070.485: [ParNew: 105342K->457K(118016K), 0.0054617 secs] 125901K->21026K(1035520K), 0.0055950 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-11T18:14:22.575-0500: 115828.056: [GC (Allocation Failure) 2018-04-11T18:14:22.576-0500: 115828.056: [ParNew: 105417K->376K(118016K), 0.0054509 secs] 125986K->20950K(1035520K), 0.0055927 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-11T20:33:05.739-0500: 124151.220: [GC (Allocation Failure) 2018-04-11T20:33:05.739-0500: 124151.220: [ParNew: 105336K->393K(118016K), 0.0051860 secs] 125910K->20974K(1035520K), 0.0053419 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-11T22:18:06.335-0500: 130451.816: [GC (Allocation Failure) 2018-04-11T22:18:06.335-0500: 130451.816: [ParNew: 105353K->311K(118016K), 0.0048221 secs] 125934K->20899K(1035520K), 0.0049536 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-12T00:14:22.577-0500: 137428.058: [GC (Allocation Failure) 2018-04-12T00:14:22.578-0500: 137428.058: [ParNew: 105271K->363K(118016K), 0.0062519 secs] 125859K->20954K(1035520K), 0.0063838 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-12T02:11:06.809-0500: 144432.290: [GC (Allocation Failure) 2018-04-12T02:11:06.809-0500: 144432.290: [ParNew: 105323K->332K(118016K), 0.0033749 secs] 125914K->20922K(1035520K), 0.0034735 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-12T04:05:07.411-0500: 151272.891: [GC (Allocation Failure) 2018-04-12T04:05:07.411-0500: 151272.891: [ParNew: 105292K->377K(118016K), 0.0053572 secs] 125882K->20975K(1035520K), 0.0054758 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-12T05:52:07.833-0500: 157693.313: [GC (Allocation Failure) 2018-04-12T05:52:07.833-0500: 157693.313: [ParNew: 105337K->319K(118016K), 0.0051428 secs] 125935K->20920K(1035520K), 0.0052699 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-12T07:47:08.070-0500: 164593.551: [GC (Allocation Failure) 2018-04-12T07:47:08.070-0500: 164593.551: [ParNew: 105279K->322K(118016K), 0.0050642 secs] 125880K->20922K(1035520K), 0.0051929 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-12T09:44:08.307-0500: 171613.788: [GC (Allocation Failure) 2018-04-12T09:44:08.307-0500: 171613.788: [ParNew: 105282K->334K(118016K), 0.0053739 secs] 125882K->20940K(1035520K), 0.0055210 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-12T11:30:08.940-0500: 177974.421: [GC (Allocation Failure) 2018-04-12T11:30:08.940-0500: 177974.421: [ParNew: 105294K->336K(118016K), 0.0046277 secs] 125900K->20943K(1035520K), 0.0047000 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-12T13:27:09.175-0500: 184994.655: [GC (Allocation Failure) 2018-04-12T13:27:09.175-0500: 184994.655: [ParNew: 105296K->307K(118016K), 0.0053687 secs] 125903K->20914K(1035520K), 0.0055060 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-12T15:23:22.545-0500: 191968.026: [GC (Allocation Failure) 2018-04-12T15:23:22.545-0500: 191968.026: [ParNew: 105267K->331K(118016K), 0.0069983 secs] 125874K->20942K(1035520K), 0.0071407 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-12T16:59:10.054-0500: 197715.535: [GC (Allocation Failure) 2018-04-12T16:59:10.054-0500: 197715.535: [ParNew: 105291K->364K(118016K), 0.0046497 secs] 125902K->20979K(1035520K), 0.0047883 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-12T18:56:10.286-0500: 204735.766: [GC (Allocation Failure) 2018-04-12T18:56:10.286-0500: 204735.766: [ParNew: 105324K->410K(118016K), 0.0049552 secs] 125939K->21036K(1035520K), 0.0050758 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-12T20:53:10.521-0500: 211756.002: [GC (Allocation Failure) 2018-04-12T20:53:10.522-0500: 211756.002: [ParNew: 105370K->407K(118016K), 0.0048750 secs] 125996K->21034K(1035520K), 0.0050050 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-12T22:18:11.304-0500: 216856.785: [GC (Allocation Failure) 2018-04-12T22:18:11.304-0500: 216856.785: [ParNew: 105367K->355K(118016K), 0.0046686 secs] 125994K->20983K(1035520K), 0.0048152 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T00:15:11.537-0500: 223877.018: [GC (Allocation Failure) 2018-04-13T00:15:11.537-0500: 223877.018: [ParNew: 105315K->437K(118016K), 0.0049622 secs] 125943K->21065K(1035520K), 0.0050967 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T01:50:11.729-0500: 229577.210: [GC (Allocation Failure) 2018-04-13T01:50:11.729-0500: 229577.210: [ParNew: 105397K->393K(118016K), 0.0048758 secs] 126025K->21021K(1035520K), 0.0050136 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-13T03:25:11.921-0500: 235277.402: [GC (Allocation Failure) 2018-04-13T03:25:11.921-0500: 235277.402: [ParNew: 105353K->313K(118016K), 0.0043602 secs] 125981K->21004K(1035520K), 0.0044876 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-13T04:51:12.476-0500: 240437.957: [GC (Allocation Failure) 2018-04-13T04:51:12.476-0500: 240437.957: [ParNew: 105273K->266K(118016K), 0.0049405 secs] 125964K->20961K(1035520K), 0.0050743 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T06:25:25.556-0500: 246091.037: [GC (Allocation Failure) 2018-04-13T06:25:25.556-0500: 246091.037: [ParNew: 105226K->291K(118016K), 0.0051227 secs] 125921K->20986K(1035520K), 0.0052423 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2018-04-13T08:00:12.850-0500: 251778.331: [GC (Allocation Failure) 2018-04-13T08:00:12.850-0500: 251778.331: [ParNew: 105251K->274K(118016K), 0.0049420 secs] 125946K->20984K(1035520K), 0.0050824 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
2018-04-13T09:35:13.039-0500: 257478.519: [GC (Allocation Failure) 2018-04-13T09:35:13.039-0500: 257478.520: [ParNew: 105234K->256K(118016K), 0.0048491 secs] 125944K->20966K(1035520K), 0.0049779 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
Heap
par new generation total 118016K, used 60774K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 57% used [0x00000000c0000000, 0x00000000c3b19838, 0x00000000c6680000)
from space 13056K, 1% used [0x00000000c7340000, 0x00000000c7380120, 0x00000000c8000000)
to space 13056K, 0% used [0x00000000c6680000, 0x00000000c6680000, 0x00000000c7340000)
concurrent mark-sweep generation total 917504K, used 20710K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 26579K, capacity 26830K, committed 27236K, reserved 1073152K
class space used 2751K, capacity 2876K, committed 2916K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201804021324 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(2191976k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T13:24:39.095-0500: 1.454: [GC (Allocation Failure) 2018-04-02T13:24:39.095-0500: 1.454: [ParNew: 104960K->11097K(118016K), 0.0324367 secs] 104960K->11097K(1035520K), 0.0325963 secs] [Times: user=0.14 sys=0.00, real=0.03 secs]
2018-04-02T13:25:40.025-0500: 62.384: [GC (Allocation Failure) 2018-04-02T13:25:40.025-0500: 62.384: [ParNew: 116057K->13056K(118016K), 0.0854199 secs] 116057K->25457K(1035520K), 0.0855136 secs] [Times: user=0.56 sys=0.03, real=0.08 secs]
2018-04-02T13:25:40.111-0500: 62.471: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12401K(917504K)] 26022K(1035520K), 0.0115051 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-04-02T13:25:40.123-0500: 62.483: [CMS-concurrent-mark-start]
2018-04-02T13:25:40.134-0500: 62.493: [CMS-concurrent-mark: 0.011/0.011 secs] [Times: user=0.08 sys=0.00, real=0.01 secs]
2018-04-02T13:25:40.134-0500: 62.493: [CMS-concurrent-preclean-start]
2018-04-02T13:25:40.137-0500: 62.496: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2018-04-02T13:25:40.137-0500: 62.497: [GC (CMS Final Remark) [YG occupancy: 14752 K (118016 K)]2018-04-02T13:25:40.137-0500: 62.497: [Rescan (parallel) , 0.0082362 secs]2018-04-02T13:25:40.145-0500: 62.505: [weak refs processing, 0.0000555 secs]2018-04-02T13:25:40.145-0500: 62.505: [class unloading, 0.0037429 secs]2018-04-02T13:25:40.149-0500: 62.509: [scrub symbol table, 0.0048198 secs]2018-04-02T13:25:40.154-0500: 62.514: [scrub string table, 0.0005803 secs][1 CMS-remark: 12401K(917504K)] 27153K(1035520K), 0.0183366 secs] [Times: user=0.08 sys=0.01, real=0.02 secs]
2018-04-02T13:25:40.156-0500: 62.515: [CMS-concurrent-sweep-start]
2018-04-02T13:25:40.158-0500: 62.517: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T13:25:40.158-0500: 62.518: [CMS-concurrent-reset-start]
2018-04-02T13:25:40.167-0500: 62.527: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
Heap
par new generation total 118016K, used 74529K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 58% used [0x00000000c0000000, 0x00000000c3c086b0, 0x00000000c6680000)
from space 13056K, 100% used [0x00000000c6680000, 0x00000000c7340000, 0x00000000c7340000)
to space 13056K, 0% used [0x00000000c7340000, 0x00000000c7340000, 0x00000000c8000000)
concurrent mark-sweep generation total 917504K, used 12396K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 25138K, capacity 25422K, committed 25676K, reserved 1073152K
class space used 2740K, capacity 2841K, committed 2892K, reserved 1048576K
==> /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-nodo1.root.com.mx.out.2 <==
ulimit -a for user hdfs
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 61064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
==> /var/log/hadoop/hdfs/gc.log-201804020944 <==
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 17 2018 16:35:30 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
Memory: 4k page, physical 15733340k(7161368k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=6 -XX:NewSize=134217728 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-04-02T09:44:15.296-0500: 1.482: [GC (Allocation Failure) 2018-04-02T09:44:15.296-0500: 1.482: [ParNew: 104960K->10951K(118016K), 0.0185539 secs] 104960K->10951K(1035520K), 0.0186699 secs] [Times: user=0.04 sys=0.02, real=0.02 secs]
2018-04-02T09:45:16.579-0500: 62.766: [GC (Allocation Failure) 2018-04-02T09:45:16.579-0500: 62.766: [ParNew: 115911K->13055K(118016K), 0.0915479 secs] 115911K->25378K(1035520K), 0.0916399 secs] [Times: user=0.57 sys=0.03, real=0.09 secs]
2018-04-02T09:45:16.671-0500: 62.858: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12322K(917504K)] 26036K(1035520K), 0.0068092 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T09:45:16.678-0500: 62.865: [CMS-concurrent-mark-start]
2018-04-02T09:45:16.687-0500: 62.874: [CMS-concurrent-mark: 0.009/0.009 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
2018-04-02T09:45:16.688-0500: 62.874: [CMS-concurrent-preclean-start]
2018-04-02T09:45:16.690-0500: 62.876: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T09:45:16.690-0500: 62.877: [GC (CMS Final Remark) [YG occupancy: 13713 K (118016 K)]2018-04-02T09:45:16.690-0500: 62.877: [Rescan (parallel) , 0.0072403 secs]2018-04-02T09:45:16.697-0500: 62.884: [weak refs processing, 0.0000487 secs]2018-04-02T09:45:16.697-0500: 62.884: [class unloading, 0.0036272 secs]2018-04-02T09:45:16.701-0500: 62.888: [scrub symbol table, 0.0048688 secs]2018-04-02T09:45:16.706-0500: 62.892: [scrub string table, 0.0005731 secs][1 CMS-remark: 12322K(917504K)] 26036K(1035520K), 0.0172773 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2018-04-02T09:45:16.708-0500: 62.894: [CMS-concurrent-sweep-start]
2018-04-02T09:45:16.710-0500: 62.896: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-04-02T09:45:16.710-0500: 62.896: [CMS-concurrent-reset-start]
2018-04-02T09:45:16.719-0500: 62.905: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2018-04-02T10:59:17.633-0500: 4503.820: [GC (Allocation Failure) 2018-04-02T10:59:17.633-0500: 4503.820: [ParNew: 118015K->3137K(118016K), 0.0376292 secs] 130332K->22383K(1035520K), 0.0377760 secs] [Times: user=0.27 sys=0.00, real=0.04 secs]
Heap
par new generation total 118016K, used 17386K [0x00000000c0000000, 0x00000000c8000000, 0x00000000c8000000)
eden space 104960K, 13% used [0x00000000c0000000, 0x00000000c0dea1e8, 0x00000000c6680000)
from space 13056K, 24% used [0x00000000c7340000, 0x00000000c7650718, 0x00000000c8000000)
to space 13056K, 0% used [0x00000000c6680000, 0x00000000c6680000, 0x00000000c7340000)
concurrent mark-sweep generation total 917504K, used 19245K [0x00000000c8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 24569K, capacity 24828K, committed 25188K, reserved 1071104K
class space used 2695K, capacity 2810K, committed 2916K, reserved 1048576K
Command failed after 1 tries
... View more
Labels:
- Labels:
-
Apache Hadoop
04-09-2018
06:05 PM
The connection test works but i fix the error only drop the database ranger, thank you for the answer.
... View more
04-06-2018
06:53 PM
Hi everyone !! I have a problem when i start the Ranger Service my logs are the following. The problem is SSL, but don't want activate it, i tried put the next propierties in ranger > advanced config but have the same error also i tried disable ssl on mysql with, thank you so much. Regards !! Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 216, in <module>
RangerAdmin().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 762, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 89, in start
self.configure(env, upgrade_type=upgrade_type, setup_db=params.stack_supports_ranger_setup_db_on_start)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 117, in locking_configure
original_configure(obj, *args, **kw)
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 130, in configure
setup_ranger_db()
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 266, in setup_ranger_db
user=params.unix_user,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-python-wrap /usr/hdp/current/ranger-admin/db_setup.py' returned 1. 2018-04-06 13:32:35,435 [I] DB FLAVOR :MYSQL
2018-04-06 13:32:35,436 [I] --------- Verifying Ranger DB connection ---------
2018-04-06 13:32:35,436 [I] Checking connection..
2018-04-06 13:32:35,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "SELECT version();"
Fri Apr 06 13:32:35 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:35,976 [I] Checking connection passed.
2018-04-06 13:32:35,976 [I] --------- Verifying version history table ---------
2018-04-06 13:32:35,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "show tables like 'x_db_version_h';"
Fri Apr 06 13:32:36 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:36,476 [I] Table x_db_version_h already exists in database 'ranger'
2018-04-06 13:32:36,476 [I] --------- Importing Ranger Core DB Schema ---------
2018-04-06 13:32:36,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
Fri Apr 06 13:32:36 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:37,024 [I] CORE_DB_SCHEMA is already imported
2018-04-06 13:32:37,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 0.7.0.2.6.4.0-91' and active='Y';"
Fri Apr 06 13:32:37 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:37,471 [I] --------- Applying Ranger DB patches ---------
2018-04-06 13:32:37,472 [I] Executing patch on ranger from file: 001-groupsource.sql
2018-04-06 13:32:37,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '001' and active = 'Y';"
Fri Apr 06 13:32:37 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:38,014 [I] Patch 001-groupsource.sql is already applied
2018-04-06 13:32:38,014 [I] Executing patch on ranger from file: 002-policyname.sql
2018-04-06 13:32:38,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '002' and active = 'Y';"
Fri Apr 06 13:32:38 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:38,543 [I] Patch 002-policyname.sql is already applied
2018-04-06 13:32:38,544 [I] Executing patch on ranger from file: 003-knoxrepo.sql
2018-04-06 13:32:38,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '003' and active = 'Y';"
Fri Apr 06 13:32:38 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:39,012 [I] Patch 003-knoxrepo.sql is already applied
2018-04-06 13:32:39,013 [I] Executing patch on ranger from file: 004-assetconfigsize.sql
2018-04-06 13:32:39,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '004' and active = 'Y';"
Fri Apr 06 13:32:39 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:39,563 [I] Patch 004-assetconfigsize.sql is already applied
2018-04-06 13:32:39,563 [I] Executing patch on ranger from file: 005-xtrxlogcolumnsize.sql
2018-04-06 13:32:39,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '005' and active = 'Y';"
Fri Apr 06 13:32:39 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:40,090 [I] Patch 005-xtrxlogcolumnsize.sql is already applied
2018-04-06 13:32:40,091 [I] Executing patch on ranger from file: 006-createdefaultpublicgroup.sql
2018-04-06 13:32:40,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '006' and active = 'Y';"
Fri Apr 06 13:32:40 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:40,569 [I] Patch 006-createdefaultpublicgroup.sql is already applied
2018-04-06 13:32:40,570 [I] Executing patch on ranger from file: 007-updateBlankPolicyName.sql
2018-04-06 13:32:40,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '007' and active = 'Y';"
Fri Apr 06 13:32:40 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:41,016 [I] Patch 007-updateBlankPolicyName.sql is already applied
2018-04-06 13:32:41,017 [I] Executing patch on ranger from file: 008-removeTrailingSlash.sql
2018-04-06 13:32:41,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '008' and active = 'Y';"
Fri Apr 06 13:32:41 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:41,461 [I] Patch 008-removeTrailingSlash.sql is already applied
2018-04-06 13:32:41,461 [I] Executing patch on ranger from file: 009-updated_schema.sql
2018-04-06 13:32:41,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '009' and active = 'Y';"
Fri Apr 06 13:32:41 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:41,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '009' and active = 'N';"
Fri Apr 06 13:32:42 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:42,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('009', now(), 'Ranger 0.7.0.2.6.4.0-91', now(), 'nodo1.root.com.mx','N') ;"
Fri Apr 06 13:32:42 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:42,946 [I] Patch 009-updated_schema.sql is being applied..
2018-04-06 13:32:42,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -input /usr/hdp/current/ranger-admin/db/mysql/patches/009-updated_schema.sql
Fri Apr 06 13:32:43 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Error executing: DROP TABLE IF EXISTS `x_service_def`;
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails
SQLException : SQL state: 23000 com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails ErrorCode: 1217
2018-04-06 13:32:43,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '009' and active = 'Y';"
Fri Apr 06 13:32:43 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:43,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "delete from x_db_version_h where version='009' and active='N' and updated_by='nodo1.root.com.mx';"
Fri Apr 06 13:32:44 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:44,399 [E] 009-updated_schema.sql import failed! stdout: /var/lib/ambari-agent/data/output-1344.txt 2018-04-06 13:32:27,649 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-04-06 13:32:28,087 - Stack Feature Version Info: stack_version=2.6, version=2.6.4.0-91, current_cluster_version=2.6.4.0-91 -> 2.6.4.0-91
2018-04-06 13:32:28,100 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-04-06 13:32:28,102 - Group['livy'] {}
2018-04-06 13:32:28,104 - Group['spark'] {}
2018-04-06 13:32:28,104 - Group['ranger'] {}
2018-04-06 13:32:28,104 - Group['hadoop'] {}
2018-04-06 13:32:28,105 - Group['users'] {}
2018-04-06 13:32:28,105 - Group['knox'] {}
2018-04-06 13:32:28,105 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,106 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,107 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,109 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,110 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,111 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-06 13:32:28,112 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,113 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-06 13:32:28,114 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']}
2018-04-06 13:32:28,115 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-06 13:32:28,116 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,117 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,118 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,119 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-06 13:32:28,120 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,120 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,121 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,122 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,123 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,124 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,124 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,125 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,126 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 13:32:28,127 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-06 13:32:28,128 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-04-06 13:32:28,136 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-04-06 13:32:28,136 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-04-06 13:32:28,137 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-06 13:32:28,138 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-04-06 13:32:28,146 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2018-04-06 13:32:28,146 - Group['hdfs'] {}
2018-04-06 13:32:28,146 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2018-04-06 13:32:28,147 - FS Type:
2018-04-06 13:32:28,147 - Directory['/etc/hadoop'] {'mode': 0755}
2018-04-06 13:32:28,166 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-06 13:32:28,167 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-04-06 13:32:28,186 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-04-06 13:32:28,198 - Skipping Execute[('setenforce', '0')] due to not_if
2018-04-06 13:32:28,199 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-04-06 13:32:28,202 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-04-06 13:32:28,202 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-04-06 13:32:28,208 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-04-06 13:32:28,211 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-04-06 13:32:28,220 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-04-06 13:32:28,237 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-06 13:32:28,238 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-04-06 13:32:28,239 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-04-06 13:32:28,246 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2018-04-06 13:32:28,254 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-04-06 13:32:28,781 - Stack Feature Version Info: stack_version=2.6, version=2.6.4.0-91, current_cluster_version=2.6.4.0-91 -> 2.6.4.0-91
2018-04-06 13:32:28,842 - Execute['/usr/bin/ranger-admin-stop'] {'environment': {'JAVA_HOME': u'/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64'}, 'user': 'ranger'}
2018-04-06 13:32:28,991 - File['/var/run/ranger/rangeradmin.pid'] {'action': ['delete']}
2018-04-06 13:32:28,991 - Temporarily skipping status check for RANGER service only.
2018-04-06 13:32:28,997 - File['/usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar'] {'action': ['delete']}
2018-04-06 13:32:28,998 - Deleting File['/usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar']
2018-04-06 13:32:28,998 - File['/var/lib/ambari-agent/tmp/mysql-connector-java.jar'] {'content': DownloadSource('http://master.root.com.mx:8080/resources//mysql-connector-java.jar'), 'mode': 0644}
2018-04-06 13:32:28,999 - Not downloading the file from http://master.root.com.mx:8080/resources//mysql-connector-java.jar, because /var/lib/ambari-agent/tmp/mysql-connector-java.jar already exists
2018-04-06 13:32:29,002 - Execute[('cp', '--remove-destination', u'/var/lib/ambari-agent/tmp/mysql-connector-java.jar', u'/usr/hdp/2.6.4.0-91/ranger-admin/ews/lib')] {'path': ['/bin', '/usr/bin/'], 'sudo': True}
2018-04-06 13:32:29,016 - File['/usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar'] {'mode': 0644}
2018-04-06 13:32:29,017 - ModifyPropertiesFile['/usr/hdp/2.6.4.0-91/ranger-admin/install.properties'] {'owner': 'ranger', 'properties': ...}
2018-04-06 13:32:29,018 - Modifying existing properties file: /usr/hdp/2.6.4.0-91/ranger-admin/install.properties
2018-04-06 13:32:29,041 - File['/usr/hdp/2.6.4.0-91/ranger-admin/install.properties'] {'owner': 'ranger', 'content': ..., 'group': None, 'mode': None, 'encoding': 'utf-8'}
2018-04-06 13:32:29,041 - Writing File['/usr/hdp/2.6.4.0-91/ranger-admin/install.properties'] because contents don't match
2018-04-06 13:32:29,042 - ModifyPropertiesFile['/usr/hdp/2.6.4.0-91/ranger-admin/install.properties'] {'owner': 'ranger', 'properties': {'SQL_CONNECTOR_JAR': u'/usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar'}}
2018-04-06 13:32:29,042 - Modifying existing properties file: /usr/hdp/2.6.4.0-91/ranger-admin/install.properties
2018-04-06 13:32:29,045 - File['/usr/hdp/2.6.4.0-91/ranger-admin/install.properties'] {'owner': 'ranger', 'content': ..., 'group': None, 'mode': None, 'encoding': 'utf-8'}
2018-04-06 13:32:29,045 - Writing File['/usr/hdp/2.6.4.0-91/ranger-admin/install.properties'] because contents don't match
2018-04-06 13:32:29,046 - ModifyPropertiesFile['/usr/hdp/current/ranger-admin/install.properties'] {'owner': 'ranger', 'properties': {'audit_store': u'solr'}}
2018-04-06 13:32:29,046 - Modifying existing properties file: /usr/hdp/current/ranger-admin/install.properties
2018-04-06 13:32:29,048 - File['/usr/hdp/current/ranger-admin/install.properties'] {'owner': 'ranger', 'content': ..., 'group': None, 'mode': None, 'encoding': 'utf-8'}
2018-04-06 13:32:29,049 - Writing File['/usr/hdp/current/ranger-admin/install.properties'] because contents don't match
2018-04-06 13:32:29,049 - Setting up Ranger DB and DB User
2018-04-06 13:32:29,050 - Execute['ambari-python-wrap /usr/hdp/current/ranger-admin/dba_script.py -q'] {'logoutput': True, 'environment': {'RANGER_ADMIN_HOME': u'/usr/hdp/current/ranger-admin', 'JAVA_HOME': u'/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64'}, 'user': 'ranger'}
2018-04-06 13:32:29,243 [I] Running DBA setup script. QuiteMode:True
2018-04-06 13:32:29,244 [I] Using Java:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java
2018-04-06 13:32:29,244 [I] DB FLAVOR:MYSQL
2018-04-06 13:32:29,244 [I] DB Host:10.1.5.20
2018-04-06 13:32:29,244 [I] ---------- Verifying DB root password ----------
2018-04-06 13:32:29,245 [I] DBA root user password validated
2018-04-06 13:32:29,245 [I] ---------- Verifying Ranger Admin db user password ----------
2018-04-06 13:32:29,245 [I] admin user password validated
2018-04-06 13:32:29,246 [I] ---------- Creating Ranger Admin db user ----------
2018-04-06 13:32:29,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "SELECT version();"
Fri Apr 06 13:32:29 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:29,792 [I] Verifying user rangeradmin for Host %
2018-04-06 13:32:29,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "select user from mysql.user where user='rangeradmin' and host='%';"
Fri Apr 06 13:32:30 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:30,333 [I] MySQL user rangeradmin already exists for host %
2018-04-06 13:32:30,333 [I] Verifying user rangeradmin for Host localhost
2018-04-06 13:32:30,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "select user from mysql.user where user='rangeradmin' and host='localhost';"
Fri Apr 06 13:32:30 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:30,890 [I] MySQL user rangeradmin already exists for host localhost
2018-04-06 13:32:30,890 [I] Verifying user rangeradmin for Host 10.1.5.20
2018-04-06 13:32:30,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "select user from mysql.user where user='rangeradmin' and host='10.1.5.20';"
Fri Apr 06 13:32:31 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:31,426 [I] MySQL user rangeradmin already exists for host 10.1.5.20
2018-04-06 13:32:31,427 [I] ---------- Creating Ranger Admin database ----------
2018-04-06 13:32:31,427 [I] Verifying database ranger
2018-04-06 13:32:31,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "show databases like 'ranger';"
Fri Apr 06 13:32:31 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:31,868 [I] Database ranger already exists.
2018-04-06 13:32:31,869 [I] ---------- Granting permission to Ranger Admin db user ----------
2018-04-06 13:32:31,869 [I] ---------- Granting privileges TO user 'rangeradmin'@'%' on db 'ranger'----------
2018-04-06 13:32:31,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "grant all privileges on ranger.* to 'rangeradmin'@'%' with grant option;"
Fri Apr 06 13:32:32 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:32,313 [I] ---------- FLUSH PRIVILEGES ----------
2018-04-06 13:32:32,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "FLUSH PRIVILEGES;"
Fri Apr 06 13:32:32 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:32,759 [I] Privileges granted to 'rangeradmin' on 'ranger'
2018-04-06 13:32:32,759 [I] ---------- Granting privileges TO user 'rangeradmin'@'localhost' on db 'ranger'----------
2018-04-06 13:32:32,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "grant all privileges on ranger.* to 'rangeradmin'@'localhost' with grant option;"
Fri Apr 06 13:32:33 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:33,350 [I] ---------- FLUSH PRIVILEGES ----------
2018-04-06 13:32:33,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "FLUSH PRIVILEGES;"
Fri Apr 06 13:32:33 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:33,897 [I] Privileges granted to 'rangeradmin' on 'ranger'
2018-04-06 13:32:33,897 [I] ---------- Granting privileges TO user 'rangeradmin'@'10.1.5.20' on db 'ranger'----------
2018-04-06 13:32:33,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "grant all privileges on ranger.* to 'rangeradmin'@'10.1.5.20' with grant option;"
Fri Apr 06 13:32:34 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:34,434 [I] ---------- FLUSH PRIVILEGES ----------
2018-04-06 13:32:34,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/mysql -u root -p '********' -noheader -trim -c \; -query "FLUSH PRIVILEGES;"
Fri Apr 06 13:32:34 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:34,902 [I] Privileges granted to 'rangeradmin' on 'ranger'
2018-04-06 13:32:34,903 [I] ---------- Ranger Policy Manager DB and User Creation Process Completed.. ----------
2018-04-06 13:32:34,919 - Execute['ambari-python-wrap /usr/hdp/current/ranger-admin/db_setup.py'] {'logoutput': True, 'environment': {'PATH': '/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent', 'RANGER_ADMIN_HOME': u'/usr/hdp/current/ranger-admin', 'JAVA_HOME': u'/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64'}, 'user': 'ranger'}
2018-04-06 13:32:35,435 [I] DB FLAVOR :MYSQL
2018-04-06 13:32:35,436 [I] --------- Verifying Ranger DB connection ---------
2018-04-06 13:32:35,436 [I] Checking connection..
2018-04-06 13:32:35,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "SELECT version();"
Fri Apr 06 13:32:35 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:35,976 [I] Checking connection passed.
2018-04-06 13:32:35,976 [I] --------- Verifying version history table ---------
2018-04-06 13:32:35,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "show tables like 'x_db_version_h';"
Fri Apr 06 13:32:36 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:36,476 [I] Table x_db_version_h already exists in database 'ranger'
2018-04-06 13:32:36,476 [I] --------- Importing Ranger Core DB Schema ---------
2018-04-06 13:32:36,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
Fri Apr 06 13:32:36 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:37,024 [I] CORE_DB_SCHEMA is already imported
2018-04-06 13:32:37,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 0.7.0.2.6.4.0-91' and active='Y';"
Fri Apr 06 13:32:37 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:37,471 [I] --------- Applying Ranger DB patches ---------
2018-04-06 13:32:37,472 [I] Executing patch on ranger from file: 001-groupsource.sql
2018-04-06 13:32:37,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '001' and active = 'Y';"
Fri Apr 06 13:32:37 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:38,014 [I] Patch 001-groupsource.sql is already applied
2018-04-06 13:32:38,014 [I] Executing patch on ranger from file: 002-policyname.sql
2018-04-06 13:32:38,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '002' and active = 'Y';"
Fri Apr 06 13:32:38 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:38,543 [I] Patch 002-policyname.sql is already applied
2018-04-06 13:32:38,544 [I] Executing patch on ranger from file: 003-knoxrepo.sql
2018-04-06 13:32:38,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '003' and active = 'Y';"
Fri Apr 06 13:32:38 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:39,012 [I] Patch 003-knoxrepo.sql is already applied
2018-04-06 13:32:39,013 [I] Executing patch on ranger from file: 004-assetconfigsize.sql
2018-04-06 13:32:39,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '004' and active = 'Y';"
Fri Apr 06 13:32:39 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:39,563 [I] Patch 004-assetconfigsize.sql is already applied
2018-04-06 13:32:39,563 [I] Executing patch on ranger from file: 005-xtrxlogcolumnsize.sql
2018-04-06 13:32:39,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '005' and active = 'Y';"
Fri Apr 06 13:32:39 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:40,090 [I] Patch 005-xtrxlogcolumnsize.sql is already applied
2018-04-06 13:32:40,091 [I] Executing patch on ranger from file: 006-createdefaultpublicgroup.sql
2018-04-06 13:32:40,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '006' and active = 'Y';"
Fri Apr 06 13:32:40 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:40,569 [I] Patch 006-createdefaultpublicgroup.sql is already applied
2018-04-06 13:32:40,570 [I] Executing patch on ranger from file: 007-updateBlankPolicyName.sql
2018-04-06 13:32:40,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '007' and active = 'Y';"
Fri Apr 06 13:32:40 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:41,016 [I] Patch 007-updateBlankPolicyName.sql is already applied
2018-04-06 13:32:41,017 [I] Executing patch on ranger from file: 008-removeTrailingSlash.sql
2018-04-06 13:32:41,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '008' and active = 'Y';"
Fri Apr 06 13:32:41 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:41,461 [I] Patch 008-removeTrailingSlash.sql is already applied
2018-04-06 13:32:41,461 [I] Executing patch on ranger from file: 009-updated_schema.sql
2018-04-06 13:32:41,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '009' and active = 'Y';"
Fri Apr 06 13:32:41 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:41,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '009' and active = 'N';"
Fri Apr 06 13:32:42 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:42,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('009', now(), 'Ranger 0.7.0.2.6.4.0-91', now(), 'nodo1.root.com.mx','N') ;"
Fri Apr 06 13:32:42 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:42,946 [I] Patch 009-updated_schema.sql is being applied..
2018-04-06 13:32:42,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -input /usr/hdp/current/ranger-admin/db/mysql/patches/009-updated_schema.sql
Fri Apr 06 13:32:43 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Error executing: DROP TABLE IF EXISTS `x_service_def`;
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails
SQLException : SQL state: 23000 com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails ErrorCode: 1217
2018-04-06 13:32:43,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = '009' and active = 'Y';"
Fri Apr 06 13:32:43 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:43,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp /usr/hdp/2.6.4.0-91/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.1.5.20/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "delete from x_db_version_h where version='009' and active='N' and updated_by='nodo1.root.com.mx';"
Fri Apr 06 13:32:44 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-04-06 13:32:44,399 [E] 009-updated_schema.sql import failed!
Command failed after 1 tries
... View more
Labels:
- Labels:
-
Apache Ranger
04-06-2018
05:10 PM
Ya quedo resuelto el problema, tuve que reinstalar los paquetes de Ranger manualmente, yum search ranger y reinstall algunos paquetes, al volver a tirar tus comandos ya me aparecia credentialbuilder, pero me aparecia un error de que no podia ejecutar Mkdir en una ruta, cuando aparece la ruta quise acceder a ella, pero me genero errores de enlaces simbolicos duplicados, rastreando los archivos a donde apuntaban, los elimine, reinstale Ranger y funciono. Muchas Gracias, tu respuesta me dio la idea.
... View more
04-06-2018
03:35 PM
Cuando pongo las rutas me dice que no existe ningun directorio, solo llega hasta "/usr/hdp/current/" y aparece un enlace simbólico de los archivos de ranger apuntando a "/usr/ hdp / 2.6.4.0-91 / ranger-tagsync" que no existe el directorio, ya que solo aparecen los demas servicios, excepto los de ranger (ranger-tagsync, ranger-admin, etc).
... View more
04-06-2018
12:16 AM
Hi Guys !! Tengo un problema con la instalacion de Ranger, realice primero la instalacion correctamente, pero tenia algunos problemas con la conexion de MYSQL pero de la desesperacion, se me ocurrio desinstalarlo y al querer volver a instalarlo, tengo errores, y realmente no he encontrado una solucion similar, el error es al instalar Ranger Tagsync. Les comparto el archivo de log. Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py", line 135, in <module>
RangerTagsync().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py", line 41, in install
ranger_credential_helper(params.tagsync_cred_lib, 'tagadmin.user.password', 'rangertagsync', params.tagsync_jceks_path)
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 627, in ranger_credential_helper
Execute(cmd, environment={'JAVA_HOME': params.java_home}, logoutput=True, sudo=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -cp '/usr/hdp/current/ranger-tagsync/lib/*' org.apache.ranger.credentialapi.buildks create tagadmin.user.password -value '[PROTECTED]' -provider jceks://file/usr/hdp/current/ranger-tagsync/conf/rangertagsync.jceks' returned 1. Error: Could not find or load main class org.apache.ranger.credentialapi.buildks stdout: /var/lib/ambari-agent/data/output-1104.txt 2018-04-05 17:10:52,460 - Stack Feature Version Info: stack_version=2.6, version=2.6.4.0-91, current_cluster_version=2.6.4.0-91 -> 2.6.4.0-91
2018-04-05 17:10:52,469 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-04-05 17:10:52,471 - Group['livy'] {}
2018-04-05 17:10:52,472 - Group['spark'] {}
2018-04-05 17:10:52,472 - Group['ranger'] {}
2018-04-05 17:10:52,473 - Group['hadoop'] {}
2018-04-05 17:10:52,473 - Group['users'] {}
2018-04-05 17:10:52,473 - Group['knox'] {}
2018-04-05 17:10:52,473 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,474 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,475 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,476 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,476 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,477 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-05 17:10:52,478 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,479 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-05 17:10:52,479 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']}
2018-04-05 17:10:52,480 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-05 17:10:52,481 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,482 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,482 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,483 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-05 17:10:52,484 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,485 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,486 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,486 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,487 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,488 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,489 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,490 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,490 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-05 17:10:52,491 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-05 17:10:52,493 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-04-05 17:10:52,501 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-04-05 17:10:52,501 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-04-05 17:10:52,502 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-05 17:10:52,504 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-04-05 17:10:52,512 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2018-04-05 17:10:52,512 - Group['hdfs'] {}
2018-04-05 17:10:52,513 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2018-04-05 17:10:52,514 - FS Type:
2018-04-05 17:10:52,514 - Directory['/etc/hadoop'] {'mode': 0755}
2018-04-05 17:10:52,539 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-05 17:10:52,540 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-04-05 17:10:52,565 - Initializing 2 repositories
2018-04-05 17:10:52,567 - Repository['HDP-2.6'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None}
2018-04-05 17:10:52,588 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.6]\nname=HDP-2.6\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0\n\npath=/\nenabled=1\ngpgcheck=0'}
2018-04-05 17:10:52,590 - Repository['HDP-UTILS-1.1.0.21'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2018-04-05 17:10:52,600 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.21]\nname=HDP-UTILS-1.1.0.21\nbaseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7\n\npath=/\nenabled=1\ngpgcheck=0'}
2018-04-05 17:10:52,601 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-04-05 17:10:52,778 - Skipping installation of existing package unzip
2018-04-05 17:10:52,778 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-04-05 17:10:52,805 - Skipping installation of existing package curl
2018-04-05 17:10:52,805 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-04-05 17:10:52,825 - Skipping installation of existing package hdp-select
2018-04-05 17:10:53,431 - Version 2.6.4.0-91 was provided as effective cluster version. Using package version 2_6_4_0_91
2018-04-05 17:10:53,433 - Package['ranger_2_6_4_0_91-admin'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-04-05 17:10:53,554 - Skipping installation of existing package ranger_2_6_4_0_91-admin
2018-04-05 17:10:53,555 - Version 2.6.4.0-91 was provided as effective cluster version. Using package version 2_6_4_0_91
2018-04-05 17:10:53,556 - Package['ranger_2_6_4_0_91-usersync'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-04-05 17:10:53,577 - Skipping installation of existing package ranger_2_6_4_0_91-usersync
2018-04-05 17:10:53,578 - Version 2.6.4.0-91 was provided as effective cluster version. Using package version 2_6_4_0_91
2018-04-05 17:10:53,579 - Package['ranger_2_6_4_0_91-tagsync'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-04-05 17:10:53,600 - Skipping installation of existing package ranger_2_6_4_0_91-tagsync
2018-04-05 17:10:53,601 - Stack Feature Version Info: stack_version=2.6, version=2.6.4.0-91, current_cluster_version=2.6.4.0-91 -> 2.6.4.0-91
2018-04-05 17:10:53,656 - Execute[(u'/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java', '-cp', u'/usr/hdp/current/ranger-tagsync/lib/*', 'org.apache.ranger.credentialapi.buildks', 'create', 'tagadmin.user.password', '-value', [PROTECTED], '-provider', u'jceks://file/usr/hdp/current/ranger-tagsync/conf/rangertagsync.jceks')] {'logoutput': True, 'environment': {'JAVA_HOME': u'/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64'}, 'sudo': True}
Error: Could not find or load main class org.apache.ranger.credentialapi.buildks
Command failed after 1 tries
... View more
Labels:
- Labels:
-
Apache Ranger
