About wonhee_choi

wonhee_choi · ‎09-06-2016

Thanks Constantin, This issue was resolved as updated. FYI, the SQOOP_HOME variable has always been empty in any user. I think somehow sqoop, as a part of a default HDP installation, is configured to locate lib files under /usr/hdp/current/sqoop-client/lib/ regardless of the variable value.

wonhee_choi · ‎09-06-2016

Thank you vranganathan. This issue is resolved. As you pointed out there was a problem with the ojdbc6.jar file itself. downloading directly via the web url of the file neither download the complete file, nor spit error. I guess this is because of the web authentication oracle requires. After manually downloading the file after authentication and transferring the file into the sqoop host did the trick. Thanks again.

wonhee_choi · ‎09-06-2016

Thanks Constantin, There is no such directory /usr/lib/sqoop/lib. I have been restarted the service as well as the entire cluster several times along the way. So it shouldn't be the problem. Postgresql driver, FYI, worked immediately after placing into the directory above without service restart. [root@ip-172-31-29-145 lib]# cd sqoop -bash: cd: sqoop: No such file or directory [root@ip-172-31-29-145 lib]# pwd /usr/lib [root@ip-172-31-29-145 lib]# cd sqoop -bash: cd: sqoop: No such file or directory Sure, I will certainly vote and accept, if there is any actual benefit in the answers. In the mean time I deeply appreciate everyone involved in the conversation.

wonhee_choi · ‎09-06-2016

Thanks vranganathan, As you can see in the below, ojdbc6.jar file has the same privilege(644 owner root) as all other preloaded drivers. I took the exact the same steps for the postgresql driver, and sqoop against postgresql works fine. only oracle doesn't work. [root@ip-172-31-29-145 lib]# ls -ltr total 16164 -rw-r--r-- 1 root root 1245776 Apr 25 18:02 snappy-java-1.0.5.jar -rw-r--r-- 1 root root 60530 Apr 25 18:02 commons-logging-1.1.1.jar -rw-r--r-- 1 root root 266289 Apr 25 18:02 commons-jexl-2.1.1.jar -rw-r--r-- 1 root root 770013 Apr 25 18:02 jackson-mapper-asl-1.9.13.jar -rw-r--r-- 1 root root 1741973 Apr 25 18:02 kite-data-mapreduce-1.0.0.jar -rw-r--r-- 1 root root 195678 Apr 25 18:02 jackson-core-2.3.1.jar -rw-r--r-- 1 root root 224135 Apr 25 18:02 ant-contrib-1.0b3.jar -rw-r--r-- 1 root root 94608 Apr 25 18:02 xz-1.0.jar -rw-r--r-- 1 root root 241259 Apr 25 18:02 commons-compress-1.4.1.jar -rw-r--r-- 1 root root 41899 Apr 25 18:03 parquet-avro-1.4.1.jar -rw-r--r-- 1 root root 902327 Apr 25 18:03 jackson-databind-2.3.1.jar -rw-r--r-- 1 root root 1015097 Apr 25 18:03 parquet-jackson-1.4.1.jar -rw-r--r-- 1 root root 2113186 Apr 25 18:03 kite-data-core-1.0.0.jar -rw-r--r-- 1 root root 108990 Apr 25 18:03 commons-io-1.4.jar -rw-r--r-- 1 root root 1774917 Apr 25 18:03 kite-data-hive-1.0.0.jar -rw-r--r-- 1 root root 10378 Apr 25 18:03 parquet-generator-1.4.1.jar -rw-r--r-- 1 root root 466100 Apr 25 18:03 parquet-format-2.0.0.jar -rw-r--r-- 1 root root 19816 Apr 25 18:03 opencsv-2.3.jar -rw-r--r-- 1 root root 706015 Apr 25 18:03 hsqldb-1.8.0.10.jar -rw-r--r-- 1 root root 36427 Apr 25 18:03 ant-eclipse-1.0-jvm1.2.jar -rw-r--r-- 1 root root 58131 Apr 25 18:03 commons-codec-1.4.jar -rw-r--r-- 1 root root 713848 Apr 25 18:03 parquet-column-1.4.1.jar -rw-r--r-- 1 root root 11359 Apr 25 18:03 parquet-common-1.4.1.jar -rw-r--r-- 1 root root 400090 Apr 25 18:03 avro-1.7.5.jar -rw-r--r-- 1 root root 152255 Apr 25 18:03 parquet-hadoop-1.4.1.jar -rw-r--r-- 1 root root 272869 Apr 25 18:03 parquet-encoding-1.4.1.jar -rw-r--r-- 1 root root 1740983 Apr 25 18:04 kite-hadoop-compatibility-1.0.0.jar -rw-r--r-- 1 root root 25477 Apr 25 18:04 slf4j-api-1.6.1.jar -rw-r--r-- 1 root root 34047 Apr 25 18:04 jackson-annotations-2.3.0.jar -rw-r--r-- 1 root root 170228 Apr 25 18:04 avro-mapred-1.7.5-hadoop2.jar -rw-r--r-- 1 root root 229445 Apr 25 18:04 jackson-core-asl-1.9.13.jar -rw-r--r-- 1 root root 29532 Apr 25 18:04 paranamer-2.3.jar -rw-r--r-- 1 root root 579785 Sep 2 14:21 postgresql-9.2-1002.jdbc4.jar -rw-r--r-- 1 root root 2123 Sep 2 14:31 ojdbc6.jar -rw-r--r-- 1 root root 20570 Sep 2 16:24 optim_items.java lrwxrwxrwx 1 root root 40 Sep 5 09:50 mysql-connector-java.jar -> /usr/share/java/mysql-connector-java.jar [root@ip-172-31-29-145 lib]#

wonhee_choi · ‎09-05-2016

As of my original post, I have already uploaded those jdbc jar file in the library directory( /usr/hdp/current/sqoop-client/lib/) and sqoop against oracle still doesn't work.

wonhee_choi · ‎09-02-2016

Thanks vranganathan, I believe /usr/hdp/current/sqoop-client/lib/, specified above is the lib directory. Is it not? This is vanilla installation of HDP2.4, and I haven't done any additional sqoop configuration. Regards,

wonhee_choi · ‎09-02-2016

Hi All, I am having an issue with sqoop against an Oracle database. HDP2.4. Oracle 11 XE in AWS RDS. [root@ip-172-31-29-111 lib]# sqoop list-tables --connect jdbc:oracle:thin:@//dbip:dbport/dbname --username myusername -password mypassword Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 16/09/02 15:23:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258 16/09/02 15:23:54 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/09/02 15:23:54 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled. 16/09/02 15:23:54 INFO manager.SqlManager: Using default fetchSize of 1000 16/09/02 15:23:54 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:287) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.OracleManager.listTables(OracleManager.java:745) at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49) at org.apache.sqoop.Sqoop.run(Sqoop.java:148) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235) at org.apache.sqoop.Sqoop.main(Sqoop.java:244) The equivalent against a PostgreSQL works fine. The below is the list of jar files under sqoop library directory. [root@ip-172-31-29-111 lib]# ls /usr/hdp/current/sqoop-client/lib/ ant-contrib-1.0b3.jar commons-jexl-2.1.1.jar jackson-mapper-asl-1.9.13.jar opencsv-2.3.jar parquet-generator-1.4.1.jar ant-eclipse-1.0-jvm1.2.jar commons-logging-1.1.1.jar kite-data-core-1.0.0.jar paranamer-2.3.jar parquet-hadoop-1.4.1.jar avro-1.7.5.jar hsqldb-1.8.0.10.jar kite-data-hive-1.0.0.jar parquet-avro-1.4.1.jar parquet-jackson-1.4.1.jar avro-mapred-1.7.5-hadoop2.jar jackson-annotations-2.3.0.jar kite-data-mapreduce-1.0.0.jar parquet-column-1.4.1.jar postgresql-9.2-1002.jdbc4.jar commons-codec-1.4.jar jackson-core-2.3.1.jar kite-hadoop-compatibility-1.0.0.jar parquet-common-1.4.1.jar slf4j-api-1.6.1.jar commons-compress-1.4.1.jar jackson-core-asl-1.9.13.jar mysql-connector-java.jar parquet-encoding-1.4.1.jar snappy-java-1.0.5.jar commons-io-1.4.jar jackson-databind-2.3.1.jar ojdbc6.jar Earlier I have downloaded and copied the oracle jdbc connector jar file from the below Oracle page. http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html Please advise the solution to avoid the error. Thanks,

wonhee_choi · ‎07-15-2016

Sorry, I hate to repeat this, but this issue came back again. I am not trying to fool anyone here. Sorry for the misleading clue above. This time no change has been made but the cluster restart. Same error. Restarting doesn't solve the problem. This issue is so intermittent so very hard to troubleshoot.

wonhee_choi · ‎07-14-2016

Sorry. I hate to repeat this, but this issue is gone. Only change I made as per HBase is the below ambari metrics parameter values following Ambari's recommendation. hbase_master_xmn_size -> 384 hbase_master_heapsize -> 1115 Upon cluster restart, Hbase came up alright.

wonhee_choi · ‎07-14-2016

Resolved. I followed the below document to install python-pip manually in another way: http://www.liquidweb.com/kb/how-to-install-pip-on-centos-7/ [root@ip-172-31-25-3 ~]# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm --2016-07-14 11:49:14-- https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm Resolving dl.fedoraproject.org... 209.132.181.25, 209.132.181.26, 209.132.181.27, ... Connecting to dl.fedoraproject.org|209.132.181.25|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 14540 (14K) [application/x-rpm] Saving to: “epel-release-latest-6.noarch.rpm” 100%[======================================================================================================================================>] 14,540 90.8K/s in 0.2s 2016-07-14 11:49:15 (90.8 KB/s) - “epel-release-latest-6.noarch.rpm” saved [14540/14540] [root@ip-172-31-25-3 ~]# ll total 20 -rw-r--r-- 1 root root 14540 Nov 6 2012 epel-release-latest-6.noarch.rpm drwxr-xr-x 3 root root 4096 May 30 13:11 guardium_work [root@ip-172-31-25-3 ~]# sudo rpm -Uvh epel-release-latest-6.noarch.rpm warning: epel-release-latest-6.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing... ########################################### [100%] 1:epel-release ########################################### [100%] [root@ip-172-31-25-3 ~]# yum -y install python-pip Loaded plugins: amazon-id, rhui-lb, security epel/metalink | 2.7 kB 00:00 epel | 4.3 kB 00:00 epel/primary_db | 5.8 MB 00:00 Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package python-pip.noarch 0:7.1.0-1.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================================================ Installing: python-pip noarch 7.1.0-1.el6 epel 1.5 M Transaction Summary ================================================================================================================================================================================ Install 1 Package(s) Total download size: 1.5 M Installed size: 6.6 M Downloading Packages: python-pip-7.1.0-1.el6.noarch.rpm | 1.5 MB 00:00 warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 Importing GPG key 0x0608B895: Userid : EPEL (6) <epel@fedoraproject.org> Package: epel-release-6-8.noarch (installed) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Warning: RPMDB altered outside of yum. Installing : python-pip-7.1.0-1.el6.noarch 1/1 Verifying : python-pip-7.1.0-1.el6.noarch 1/1 Installed: python-pip.noarch 0:7.1.0-1.el6 Complete! [root@ip-172-31-25-3 ~]# After installing python-pip via yum, reinstalled zeppelin and it went fine. Hope this helps.

wonhee_choi · ‎07-14-2016

Hi All, Zeppelin installation via Ambari fails on HDP2.4. RHEL6.5 64bit. Below are the installation logs from Ambari. The yum command in the log actually fails when run manually. Manually installed pip as below but when uninstalled and installed again the same error. [root@ip-172-31-24-221 ~]# curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1488k 100 1488k 0 0 19.1M 0 --:--:-- --:--:-- --:--:-- 45.4M [root@ip-172-31-24-221 ~]# [root@ip-172-31-24-221 ~]# python get-pip.py DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6 Collecting pip /tmp/tmpfrM4hF/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning. /tmp/tmpfrM4hF/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB) 100% |████████████████████████████████| 1.2MB 978kB/s Collecting wheel Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB) 100% |████████████████████████████████| 71kB 9.6MB/s Collecting argparse; python_version == "2.6" (from wheel) Downloading argparse-1.4.0-py2.py3-none-any.whl Installing collected packages: pip, argparse, wheel Found existing installation: argparse 1.2.1 Uninstalling argparse-1.2.1: Successfully uninstalled argparse-1.2.1 Successfully installed argparse-1.4.0 pip-8.1.2 wheel-0.29.0 /tmp/tmpfrM4hF/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. [root@ip-172-31-24-221 ~]# pip -V pip 8.1.2 from /usr/lib/python2.6/site-packages (python 2.6) How could I get around this problem and install Zeppelin please? Plus, after this failure, starting all services in Ambari stopped working. I have to start/stop each service at a time. It is not clear whether Zeppelin install failure somehow affect this. Please advise. Thanks, Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package/scripts/master.py", line 235, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package/scripts/master.py", line 54, in install self.install_packages(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 404, in install_packages Package(name) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 49, in action_install self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install python-pip' returned 1. Error: Nothing to do 2016-07-14 10:18:17,167 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.2.0-258 2016-07-14 10:18:17,167 - Checking if need to create versioned conf dir /etc/hadoop/2.4.2.0-258/0 2016-07-14 10:18:17,167 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.2.0-258 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-07-14 10:18:17,189 - call returned (1, '/etc/hadoop/2.4.2.0-258/0 exist already', '') 2016-07-14 10:18:17,189 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.2.0-258 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-07-14 10:18:17,210 - checked_call returned (0, '') 2016-07-14 10:18:17,211 - Ensuring that hadoop has the correct symlink structure 2016-07-14 10:18:17,211 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-07-14 10:18:17,212 - Group['spark'] {} 2016-07-14 10:18:17,214 - Group['zeppelin'] {} 2016-07-14 10:18:17,214 - Adding group Group['zeppelin'] 2016-07-14 10:18:17,238 - Group['hadoop'] {} 2016-07-14 10:18:17,239 - Group['users'] {} 2016-07-14 10:18:17,239 - Group['knox'] {} 2016-07-14 10:18:17,239 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,240 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,241 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,241 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2016-07-14 10:18:17,242 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,243 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,243 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2016-07-14 10:18:17,244 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2016-07-14 10:18:17,244 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,245 - Adding user User['zeppelin'] 2016-07-14 10:18:17,296 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,297 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2016-07-14 10:18:17,297 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,298 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,299 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,299 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,300 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,301 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,301 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,302 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-07-14 10:18:17,303 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-07-14 10:18:17,305 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2016-07-14 10:18:17,309 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2016-07-14 10:18:17,309 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'} 2016-07-14 10:18:17,310 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-07-14 10:18:17,311 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2016-07-14 10:18:17,314 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if 2016-07-14 10:18:17,315 - Group['hdfs'] {} 2016-07-14 10:18:17,315 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']} 2016-07-14 10:18:17,316 - Directory['/etc/hadoop'] {'mode': 0755} 2016-07-14 10:18:17,331 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2016-07-14 10:18:17,332 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777} 2016-07-14 10:18:17,345 - Repository['HDP-2.4'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.2.0', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None} 2016-07-14 10:18:17,354 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.4]\nname=HDP-2.4\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.2.0\n\npath=/\nenabled=1\ngpgcheck=0'} 2016-07-14 10:18:17,354 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None} 2016-07-14 10:18:17,358 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.20]\nname=HDP-UTILS-1.1.0.20\nbaseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6\n\npath=/\nenabled=1\ngpgcheck=0'} 2016-07-14 10:18:17,358 - Package['unzip'] {} 2016-07-14 10:18:17,460 - Skipping installation of existing package unzip 2016-07-14 10:18:17,460 - Package['curl'] {} 2016-07-14 10:18:17,499 - Skipping installation of existing package curl 2016-07-14 10:18:17,499 - Package['hdp-select'] {} 2016-07-14 10:18:17,537 - Skipping installation of existing package hdp-select 2016-07-14 10:18:17,699 - Execute['find /var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package -iname "*.sh" | xargs chmod +x'] {} 2016-07-14 10:18:17,708 - Execute['echo platform.linux_distribution:Red Hat Enterprise Linux Server+6.5+Santiago'] {} 2016-07-14 10:18:17,712 - Package['gcc-gfortran'] {} 2016-07-14 10:18:17,812 - Installing package gcc-gfortran ('/usr/bin/yum -d 0 -e 0 -y install gcc-gfortran') 2016-07-14 10:18:22,975 - Package['blas-devel'] {} 2016-07-14 10:18:23,016 - Installing package blas-devel ('/usr/bin/yum -d 0 -e 0 -y install blas-devel') 2016-07-14 10:18:25,378 - Package['lapack-devel'] {} 2016-07-14 10:18:25,418 - Installing package lapack-devel ('/usr/bin/yum -d 0 -e 0 -y install lapack-devel') 2016-07-14 10:18:29,415 - Package['python-devel'] {} 2016-07-14 10:18:29,458 - Skipping installation of existing package python-devel 2016-07-14 10:18:29,459 - Package['python-pip'] {} 2016-07-14 10:18:29,497 - Installing package python-pip ('/usr/bin/yum -d 0 -e 0 -y install python-pip')

wonhee_choi · ‎07-13-2016

Hi sjiang, The error message in the jira post seems to be similar to the above. Yes, the file exists in the path. Two points from me: 1) We don't have any secondary/backup hbase master in place. Only 1 master and 2 region servers. Does the above jira case still apply to mine? 2) If yes, is there any workaround just to get it started? We don't really care about the data integrity here because whatever data in hbase now, is not what we have created explicitly. Cluster restarting doesn't resolve this. It always fails to start up at the moment. Thanks!

wonhee_choi · ‎07-12-2016

Hi All, This issue came back. We don't explicitly use Hbase in our environment so this intermittent hbase start-up issue is quite puzzling. We don't have any data we explicitly stored in habse. Below is the full log of /var/log/hbase/hbase-hbase-master-xxx.log at the time of initial start-up and failure. HDP 2.4 Hbase 1.1.2.2.4 Tue Jul 12 10:20:51 AEST 2016 Starting master on ip-172-31-25-3.ap-southeast-2.compute.internal core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127424 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 16000 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 2016-07-12 10:20:52,975 INFO [main] util.VersionInfo: HBase 1.1.2.2.4.2.0-258 2016-07-12 10:20:52,976 INFO [main] util.VersionInfo: Source code repository file:///grid/0/jenkins/workspace/HDP-build-centos6/bigtop/build/hbase/rpm/BUILD/hbase-1.1.2.2.4.2.0 revision=Unknown 2016-07-12 10:20:52,976 INFO [main] util.VersionInfo: Compiled by jenkins on Mon Apr 25 06:36:21 UTC 2016 2016-07-12 10:20:52,976 INFO [main] util.VersionInfo: From source with checksum 4f661ee4f9f148ce7bfcad5b0d667c27 2016-07-12 10:20:53,424 INFO [main] util.ServerCommandLine: env:PATH=/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent 2016-07-12 10:20:53,424 INFO [main] util.ServerCommandLine: env:HISTCONTROL=ignoredups 2016-07-12 10:20:53,424 INFO [main] util.ServerCommandLine: env:HBASE_PID_DIR=/var/run/hbase 2016-07-12 10:20:53,424 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVER_OPTS= -Xmn200m -XX:CMSInitiatingOccupancyFraction=70 -Xms1024m -Xmx1024m 2016-07-12 10:20:53,424 INFO [main] util.ServerCommandLine: env:HBASE_CONF_DIR=/usr/hdp/current/hbase-master/conf 2016-07-12 10:20:53,424 INFO [main] util.ServerCommandLine: env:MAIL=/var/spool/mail/hbase 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=::/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:LOGNAME=hbase 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HBASE_REST_OPTS= 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:PWD=/home/hbase 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:LESSOPEN=|/usr/bin/lesspipe.sh %s 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:SHELL=/bin/bash 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:QTINC=/usr/lib64/qt-3.3/include 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HBASE_MASTER_OPTS= -Xmx1024m 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=false 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVERS=/usr/hdp/current/hbase-master/conf/regionservers 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HADOOP_HOME=/usr/hdp/2.4.2.0-258/hadoop 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HBASE_NICENESS=0 2016-07-12 10:20:53,425 INFO [main] util.ServerCommandLine: env:HBASE_OPTS=-Dhdp.version=2.4.2.0-258 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201607121020 -Xmx1024m -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-master-ip-172-31-25-3.ap-southeast-2.compute.internal.log -Dhbase.home.dir=/usr/hdp/current/hbase-master/bin/.. -Dhbase.id.str=hbase -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:HBASE_START_FILE=/var/run/hbase/hbase-hbase-master.autorestart 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:SHLVL=3 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:ZOOKEEPER_HOME=/usr/hdp/2.4.2.0-258/zookeeper 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-hbase-master-ip-172-31-25-3.ap-southeast-2.compute.internal.log 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:HISTSIZE=1000 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:JAVA_HOME=/usr/jdk64/jdk1.8.0_60 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:HDP_VERSION=2.4.2.0-258 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:LANG=en_US.UTF-8 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:G_BROKEN_FILENAMES=1 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:HBASE_CLASSPATH=/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/*:/usr/hdp/2.4.2.0-258/hadoop/lib/*:/usr/hdp/2.4.2.0-258/zookeeper/*:/usr/hdp/2.4.2.0-258/zookeeper/lib/*: 2016-07-12 10:20:53,426 INFO [main] util.ServerCommandLine: env:CVS_RSH=ssh 2016-07-12 10:20:53,427 INFO [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=hbase 2016-07-12 10:20:53,427 INFO [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/var/run/hbase/hbase-hbase-master.znode 2016-07-12 10:20:53,427 INFO [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-hbase-master-ip-172-31-25-3.ap-southeast-2.compute.internal 2016-07-12 10:20:53,427 INFO [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/var/log/hbase 2016-07-12 10:20:53,427 INFO [main] util.ServerCommandLine: env:USER=hbase 2016-07-12 10:20:53,427 INFO [main] util.ServerCommandLine: env:CLASSPATH=/usr/hdp/current/hbase-master/conf:/usr/jdk64/jdk1.8.0_60/lib/tools.jar:/usr/hdp/current/hbase-master/bin/..:/usr/hdp/current/hbase-master/bin/../lib/activation-1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/aopalliance-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/asm-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/avro-1.7.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-cli-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-codec-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-collections-3.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-compress-1.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-configuration-1.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-digester-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-el-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-httpclient-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-io-2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-lang-2.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-logging-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math-2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math3-3.1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-net-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-client-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-framework-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/disruptor-3.3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/current/hbase-master/bin/../lib/gson-2.2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/guava-12.0.1.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-servlet-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift.jar:/usr/hdp/current/hbase-master/bin/../lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hbase-master/bin/../lib/httpclient-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/httpcore-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-2.2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jamon-runtime-2.3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-compiler-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-runtime-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/javax.inject-1.jar:/usr/hdp/current/hbase-master/bin/../lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hbase-master/bin/../lib/jcodings-1.0.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-client-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-core-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-guice-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-json-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-server-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jets3t-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/jettison-1.3.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/joni-2.1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jruby-complete-1.6.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jsch-0.1.42.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsr305-1.3.9.jar:/usr/hdp/current/hbase-master/bin/../lib/junit-4.11.jar:/usr/hdp/current/hbase-master/bin/../lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/libthrift-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/log4j-1.2.17.jar:/usr/hdp/current/hbase-master/bin/../lib/metrics-core-2.2.0.jar:/usr/hdp/current/hbase-master/bin/../lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-3.2.4.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/ojdbc6.jar:/usr/hdp/current/hbase-master/bin/../lib/okhttp-2.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/okio-1.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/paranamer-2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-hbase-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-plugin-classloader-0.5.0.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/slf4j-api-1.7.7.jar:/usr/hdp/current/hbase-master/bin/../lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/spymemcached-2.11.6.jar:/usr/hdp/current/hbase-master/bin/../lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hbase-master/bin/../lib/xml-apis-1.3.04.jar:/usr/hdp/current/hbase-master/bin/../lib/xmlenc-0.52.jar:/usr/hdp/current/hbase-master/bin/../lib/xz-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/zookeeper.jar:/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/lib/*:/usr/hdp/2.4.2.0-258/hadoop/.//*:/usr/hdp/2.4.2.0-258/hadoop-hdfs/./:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/*:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//*:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/*:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//*:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/*:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//*::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.4.2.0-258/tez/*:/usr/hdp/2.4.2.0-258/tez/lib/*:/usr/hdp/2.4.2.0-258/tez/conf:/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/*:/usr/hdp/2.4.2.0-258/hadoop/lib/*:/usr/hdp/2.4.2.0-258/zookeeper/*:/usr/hdp/2.4.2.0-258/zookeeper/lib/*: 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:SERVER_GC_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201607121020 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:HADOOP_CONF=/usr/hdp/2.4.2.0-258/hadoop/conf 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:HOSTNAME=ip-172-31-25-3.ap-southeast-2.compute.internal 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:QTDIR=/usr/lib64/qt-3.3 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS= 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:HBASE_HOME=/usr/hdp/current/hbase-master/bin/.. 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:QTLIB=/usr/lib64/qt-3.3/lib 2016-07-12 10:20:53,429 INFO [main] util.ServerCommandLine: env:HOME=/home/hbase 2016-07-12 10:20:53,430 INFO [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4 2016-07-12 10:20:53,432 INFO [main] util.ServerCommandLine: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.60-b23 2016-07-12 10:20:53,433 INFO [main] util.ServerCommandLine: vmInputArguments=[-Dproc_master, -XX:OnOutOfMemoryError=kill -9 %p, -Dhdp.version=2.4.2.0-258, -XX:+UseConcMarkSweepGC, -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log, -Djava.io.tmpdir=/tmp, -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -Xloggc:/var/log/hbase/gc.log-201607121020, -Xmx1024m, -Dhbase.log.dir=/var/log/hbase, -Dhbase.log.file=hbase-hbase-master-ip-172-31-25-3.ap-southeast-2.compute.internal.log, -Dhbase.home.dir=/usr/hdp/current/hbase-master/bin/.., -Dhbase.id.str=hbase, -Dhbase.root.logger=INFO,RFA, -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native, -Dhbase.security.logger=INFO,RFAS] 2016-07-12 10:20:54,405 INFO [main] regionserver.RSRpcServices: master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000 server-side HConnection retries=350 2016-07-12 10:20:54,559 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2016-07-12 10:20:54,586 INFO [main] ipc.RpcServer: master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000: started 10 reader(s). 2016-07-12 10:20:54,667 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2016-07-12 10:20:54,700 INFO [main] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2016-07-12 10:20:54,701 INFO [main] timeline.HadoopTimelineMetricsSink: Identified hostname = ip-172-31-25-3.ap-southeast-2.compute.internal, serviceName = hbase 2016-07-12 10:20:54,706 INFO [main] timeline.HadoopTimelineMetricsSink: Collector Uri: http://ip-172-31-25-3.ap-southeast-2.compute.internal:6188/ws/v1/timeline/metrics 2016-07-12 10:20:54,719 INFO [main] impl.MetricsSinkAdapter: Sink timeline started 2016-07-12 10:20:54,795 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s). 2016-07-12 10:20:54,795 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2016-07-12 10:20:55,890 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-07-12 10:20:56,245 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=master:16000 connecting to ZooKeeper ensemble=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 2016-07-12 10:20:56,260 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-258--1, built on 04/25/2016 05:22 GMT 2016-07-12 10:20:56,260 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=ip-172-31-25-3.ap-southeast-2.compute.internal 2016-07-12 10:20:56,260 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_60 2016-07-12 10:20:56,260 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2016-07-12 10:20:56,260 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/jdk64/jdk1.8.0_60/jre 2016-07-12 10:20:56,260 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/usr/hdp/current/hbase-master/conf:/usr/jdk64/jdk1.8.0_60/lib/tools.jar:/usr/hdp/current/hbase-master/bin/..:/usr/hdp/current/hbase-master/bin/../lib/activation-1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/aopalliance-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/asm-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/avro-1.7.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-cli-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-codec-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-collections-3.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-compress-1.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-configuration-1.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-digester-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-el-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-httpclient-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-io-2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-lang-2.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-logging-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math-2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math3-3.1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-net-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-client-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-framework-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/disruptor-3.3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/current/hbase-master/bin/../lib/gson-2.2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/guava-12.0.1.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-servlet-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift.jar:/usr/hdp/current/hbase-master/bin/../lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hbase-master/bin/../lib/httpclient-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/httpcore-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-2.2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jamon-runtime-2.3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-compiler-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-runtime-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/javax.inject-1.jar:/usr/hdp/current/hbase-master/bin/../lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hbase-master/bin/../lib/jcodings-1.0.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-client-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-core-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-guice-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-json-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-server-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jets3t-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/jettison-1.3.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/joni-2.1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jruby-complete-1.6.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jsch-0.1.42.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsr305-1.3.9.jar:/usr/hdp/current/hbase-master/bin/../lib/junit-4.11.jar:/usr/hdp/current/hbase-master/bin/../lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/libthrift-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/log4j-1.2.17.jar:/usr/hdp/current/hbase-master/bin/../lib/metrics-core-2.2.0.jar:/usr/hdp/current/hbase-master/bin/../lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-3.2.4.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/ojdbc6.jar:/usr/hdp/current/hbase-master/bin/../lib/okhttp-2.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/okio-1.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/paranamer-2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-hbase-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-plugin-classloader-0.5.0.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/slf4j-api-1.7.7.jar:/usr/hdp/current/hbase-master/bin/../lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/spymemcached-2.11.6.jar:/usr/hdp/current/hbase-master/bin/../lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hbase-master/bin/../lib/xml-apis-1.3.04.jar:/usr/hdp/current/hbase-master/bin/../lib/xmlenc-0.52.jar:/usr/hdp/current/hbase-master/bin/../lib/xz-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/zookeeper.jar:/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ojdbc6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/activation-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/junit-4.11.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-yarn-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/spark-yarn-shuffle.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-aws.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-azure-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-annotations.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-azure.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-common.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-nfs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-common-tests.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-aws-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-common-2.7.1.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-annotations-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-nfs.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-auth.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/./:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jsp-api-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/gson-2.2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/fst-2.24.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-digester-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/objenesis-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jsch-0.1.42.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-net-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/zookeeper-3.4.6.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-api-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-client-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-registry-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-archives-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-distcp-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-sls.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-extras-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jettison-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-digester-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-extras.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-sls-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jsch-0.1.42.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//httpclient-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//activation-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//joda-time-2.9.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//junit-4.11.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-streaming-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-archives.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-rumen-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-ant-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-datajoin-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-core-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-gridmix-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-ant.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-openstack-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-auth.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//httpcore-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-lang3-3.3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//curator-framework-2.7.1.jar::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.4.2.0-258/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-examples-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-history-parser-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-runtime-internals-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-mapreduce-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-api-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-runtime-library-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-tests-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-yarn-timeline-history-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-dag-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-common-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-yarn-server-timeline-plugins-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-azure-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-mapreduce-client-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-yarn-server-web-proxy-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-aws-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-mapreduce-client-core-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.4.2.0-258/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.4.2.0-258/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-annotations-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/tez/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.4.2.0-258/tez/conf:/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/hadoop-aws.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-azure-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-annotations.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-azure.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-common.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-nfs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-common-tests.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-aws-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-annotations-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-nfs.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-auth.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ojdbc6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/activation-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/junit-4.11.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-yarn-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/spark-yarn-shuffle.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/zookeeper.jar:/usr/hdp/2.4.2.0-258/zookeeper/zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-http-shared4-2.4.jar: 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.29.2.el6.x86_64 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hbase 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hbase 2016-07-12 10:20:56,280 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/hbase 2016-07-12 10:20:56,282 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 sessionTimeout=90000 watcher=master:160000x0, quorum=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181, baseZNode=/hbase-unsecure 2016-07-12 10:20:56,302 INFO [main-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181. Will not attempt to authenticate using SASL (unknown error) 2016-07-12 10:20:56,317 INFO [main-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, initiating session 2016-07-12 10:20:56,332 INFO [main-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, sessionid = 0x155dc78eeb40000, negotiated timeout = 40000 2016-07-12 10:20:56,422 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting 2016-07-12 10:20:56,431 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: starting 2016-07-12 10:20:57,088 INFO [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-07-12 10:20:57,095 INFO [main] http.HttpRequestLog: Http request log for http.requests.master is not defined 2016-07-12 10:20:57,171 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2016-07-12 10:20:57,174 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2016-07-12 10:20:57,174 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2016-07-12 10:20:57,174 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2016-07-12 10:20:57,235 INFO [main] http.HttpServer: Jetty bound to port 16010 2016-07-12 10:20:57,235 INFO [main] mortbay.log: jetty-6.1.26.hwx 2016-07-12 10:20:58,271 INFO [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:16010 2016-07-12 10:20:58,281 INFO [main] master.HMaster: hbase.rootdir=hdfs://ip-172-31-25-3.ap-southeast-2.compute.internal:8020/apps/hbase/data, hbase.cluster.distributed=true 2016-07-12 10:20:58,324 INFO [main] master.HMaster: Adding backup master ZNode /hbase-unsecure/backup-masters/ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 2016-07-12 10:20:58,540 INFO [ip-172-31-25-3:16000.activeMasterManager] master.ActiveMasterManager: Deleting ZNode for /hbase-unsecure/backup-masters/ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 from backup master directory 2016-07-12 10:20:58,555 INFO [ip-172-31-25-3:16000.activeMasterManager] master.ActiveMasterManager: Registered Active Master=ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 2016-07-12 10:20:58,711 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x570673c0 connecting to ZooKeeper ensemble=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 2016-07-12 10:20:58,723 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 sessionTimeout=90000 watcher=hconnection-0x570673c00x0, quorum=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181, baseZNode=/hbase-unsecure 2016-07-12 10:20:58,744 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181. Will not attempt to authenticate using SASL (unknown error) 2016-07-12 10:20:58,745 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, initiating session 2016-07-12 10:20:58,752 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, sessionid = 0x155dc78eeb40001, negotiated timeout = 40000 2016-07-12 10:20:58,822 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: ClusterId : 1e143dd7-84d3-4e4c-ab04-e9bc79728b2f 2016-07-12 10:20:59,725 INFO [ip-172-31-25-3:16000.activeMasterManager] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-07-12 10:20:59,753 INFO [ip-172-31-25-3:16000.activeMasterManager] coordination.SplitLogManagerCoordination: found orphan task WALs%2Fip-172-31-29-145.ap-southeast-2.compute.internal%2C16020%2C1467338848574-splitting%2Fip-172-31-29-145.ap-southeast-2.compute.internal%252C16020%252C1467338848574..meta.1467339405366.meta 2016-07-12 10:20:59,755 INFO [ip-172-31-25-3:16000.activeMasterManager] coordination.SplitLogManagerCoordination: Found 1 orphan tasks and 0 rescan nodes 2016-07-12 10:20:59,759 INFO [main-EventThread] coordination.SplitLogManagerCoordination: creating orphan task /hbase-unsecure/splitWAL/WALs%2Fip-172-31-29-145.ap-southeast-2.compute.internal%2C16020%2C1467338848574-splitting%2Fip-172-31-29-145.ap-southeast-2.compute.internal%252C16020%252C1467338848574..meta.1467339405366.meta 2016-07-12 10:20:59,759 INFO [main-EventThread] coordination.SplitLogManagerCoordination: resubmitting unassigned orphan task /hbase-unsecure/splitWAL/WALs%2Fip-172-31-29-145.ap-southeast-2.compute.internal%2C16020%2C1467338848574-splitting%2Fip-172-31-29-145.ap-southeast-2.compute.internal%252C16020%252C1467338848574..meta.1467339405366.meta 2016-07-12 10:20:59,760 INFO [main-EventThread] coordination.SplitLogManagerCoordination: resubmitting task /hbase-unsecure/splitWAL/WALs%2Fip-172-31-29-145.ap-southeast-2.compute.internal%2C16020%2C1467338848574-splitting%2Fip-172-31-29-145.ap-southeast-2.compute.internal%252C16020%252C1467338848574..meta.1467339405366.meta 2016-07-12 10:20:59,784 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-unsecure/splitWAL/RESCAN0000061370 entered state: DONE ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 2016-07-12 10:20:59,999 INFO [ip-172-31-25-3:16000.activeMasterManager] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x45c82203 connecting to ZooKeeper ensemble=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 2016-07-12 10:20:59,999 INFO [ip-172-31-25-3:16000.activeMasterManager] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 sessionTimeout=90000 watcher=hconnection-0x45c822030x0, quorum=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181, baseZNode=/hbase-unsecure 2016-07-12 10:21:00,022 INFO [ip-172-31-25-3:16000.activeMasterManager-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181. Will not attempt to authenticate using SASL (unknown error) 2016-07-12 10:21:00,022 INFO [ip-172-31-25-3:16000.activeMasterManager-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, initiating session 2016-07-12 10:21:00,028 INFO [ip-172-31-25-3:16000.activeMasterManager-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, sessionid = 0x155dc78eeb40002, negotiated timeout = 40000 2016-07-12 10:21:00,057 INFO [ip-172-31-25-3:16000.activeMasterManager] balancer.StochasticLoadBalancer: loading config 2016-07-12 10:21:00,201 INFO [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Server active/primary master=ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815, sessionid=0x155dc78eeb40000, setting cluster-up flag (Was=true) 2016-07-12 10:21:00,237 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase-unsecure/flush-table-proc/acquired /hbase-unsecure/flush-table-proc/reached /hbase-unsecure/flush-table-proc/abort 2016-07-12 10:21:00,250 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase-unsecure/online-snapshot/acquired /hbase-unsecure/online-snapshot/reached /hbase-unsecure/online-snapshot/abort 2016-07-12 10:21:00,305 INFO [ip-172-31-25-3:16000.activeMasterManager] master.MasterCoprocessorHost: System coprocessor loading is enabled 2016-07-12 10:21:00,369 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure2.ProcedureExecutor: Starting procedure executor threads=5 2016-07-12 10:21:00,370 INFO [ip-172-31-25-3:16000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery 2016-07-12 10:21:00,378 INFO [ip-172-31-25-3:16000.activeMasterManager] util.FSHDFSUtils: Recovering lease on dfs file hdfs://ip-172-31-25-3.ap-southeast-2.compute.internal:8020/apps/hbase/data/MasterProcWALs/state-############00000379.log 2016-07-12 10:21:00,402 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Failed to become active master java.io.FileNotFoundException: File does not exist: /apps/hbase/data/MasterProcWALs/state-############00000379.log at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1261) at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:280) at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:276) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:276) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverLease(FSHDFSUtils.java:282) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:215) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162) at org.apache.hadoop.hbase.master.procedure.MasterProcedureEnv$WALStoreLeaseRecovery.recoverFileLease(MasterProcedureEnv.java:58) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:925) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:276) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:430) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1223) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1122) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:709) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:193) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1793) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /apps/hbase/data/MasterProcWALs/state-############00000379.log at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200) at org.apache.hadoop.ipc.Client.call(Client.java:1426) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy16.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:605) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy17.recoverLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1259) ... 17 more 2016-07-12 10:21:00,409 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: [] 2016-07-12 10:21:00,409 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown. java.io.FileNotFoundException: File does not exist: /apps/hbase/data/MasterProcWALs/state-############00000379.log at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1261) at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:280) at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:276) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:276) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverLease(FSHDFSUtils.java:282) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:215) at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162) at org.apache.hadoop.hbase.master.procedure.MasterProcedureEnv$WALStoreLeaseRecovery.recoverFileLease(MasterProcedureEnv.java:58) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:925) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:276) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:430) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1223) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1122) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:709) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:193) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1793) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /apps/hbase/data/MasterProcWALs/state-############00000379.log at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200) at org.apache.hadoop.ipc.Client.call(Client.java:1426) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy16.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:605) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy17.recoverLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1259) ... 17 more 2016-07-12 10:21:00,410 INFO [ip-172-31-25-3:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown. 2016-07-12 10:21:00,410 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: Stopping infoServer 2016-07-12 10:21:00,421 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010 2016-07-12 10:21:00,535 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] procedure2.ProcedureExecutor: Stopping the procedure executor 2016-07-12 10:21:00,535 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store 2016-07-12 10:21:00,536 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: stopping server ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 2016-07-12 10:21:00,536 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x155dc78eeb40001 2016-07-12 10:21:00,540 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.ZooKeeper: Session: 0x155dc78eeb40001 closed 2016-07-12 10:21:00,540 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down 2016-07-12 10:21:00,541 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: stopping server ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815; all regions closed. 2016-07-12 10:21:00,541 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] hbase.ChoreService: Chore service for: ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 had [] on shutdown 2016-07-12 10:21:00,552 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x155dc78eeb40002 2016-07-12 10:21:00,556 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.ZooKeeper: Session: 0x155dc78eeb40002 closed 2016-07-12 10:21:00,556 INFO [ip-172-31-25-3:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down 2016-07-12 10:21:00,557 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] hbase.ChoreService: Chore service for: ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2016-07-12 10:21:00,557 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down. 2016-07-12 10:21:00,557 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] ipc.RpcServer: Stopping server on 16000 2016-07-12 10:21:00,557 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping 2016-07-12 10:21:00,558 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped 2016-07-12 10:21:00,560 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping 2016-07-12 10:21:00,569 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/rs/ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 already deleted, retry=false 2016-07-12 10:21:00,573 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.ZooKeeper: Session: 0x155dc78eeb40000 closed 2016-07-12 10:21:00,573 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2016-07-12 10:21:00,573 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: stopping server ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815; zookeeper connection closed. 2016-07-12 10:21:00,573 INFO [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000 exiting Please advise. Many Thanks, @sjiang @Ted Yu

wonhee_choi · ‎06-15-2016

Thanks Ted and sjiang. This issue has gone without any action - not that I know of. I have stopped the entire cluster since the issue, and when restarted it after a while hbase came up alright. No actions taken. I will reply to your answers(so you can be notified) the full log if the issue comes back. Again thanks.

wonhee_choi · ‎06-15-2016

Hi All, HBase server fails to start with the error below. A resolution suggested in another post of Hbase start issue - Deleting files under WALS path didn't help. IF you could advise what can be done or investigated that would be a great help. Regards, ----------------------------- 2016-06-15 15:02:38,692 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Failed to become active master java.io.FileNotFoundException: File does not exist: /apps/hbase/data/MasterProcWALs/state-************00000001.log 2016-06-15 15:02:38,636 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase-unsecure/flush-table-proc/acquired /hbase-unsecure/flush-table-proc/reached /hbase-unsecure/flush-table-proc/abort 2016-06-15 15:02:38,643 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase-unsecure/online-snapshot/acquired /hbase-unsecure/online-snapshot/reached /hbase-unsecure/online-snapshot/abort 2016-06-15 15:02:38,664 INFO [ip-172-31-25-3:16000.activeMasterManager] master.MasterCoprocessorHost: System coprocessor loading is enabled 2016-06-15 15:02:38,676 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure2.ProcedureExecutor: Starting procedure executor threads=5 2016-06-15 15:02:38,677 INFO [ip-172-31-25-3:16000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery 2016-06-15 15:02:38,680 INFO [ip-172-31-25-3:16000.activeMasterManager] util.FSHDFSUtils: Recovering lease on dfs file hdfs://ip-172-31-25-3.ap-southeast-2.compute.internal:8020/apps/hbase/data/MasterProcWALs/state-************00000001.log 2016-06-15 15:02:38,692 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Failed to become active master java.io.FileNotFoundException: File does not exist: /apps/hbase/data/MasterProcWALs/state-************00000001.log at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)

wonhee_choi · ‎05-17-2016

Thanks Josh, In the above case there was no data to preserve but I will be aware of the data loss by removing WAL files. I don't understand then why this has happened in the first place. Can umounting and mounting HDFS data directory to the new directory, or forceful shutdown of the cluster hosts cause this? I believe that this could be investigated separately. Much appreciated.

wonhee_choi · ‎05-13-2016

This did the trick. After removing WAL files and restarting HBase components, HBase master doesn't go down anymore. I took some time to double check everything before accepting the answer. Thank you so much.

wonhee_choi · ‎05-13-2016

Hello, HBase master wouldn't start spitting the below error. 2016-05-13 03:56:53,431 FATAL [ip-172-31-21-148:16000.activeMasterManager] master.HMaster: Failed to become active master java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned It appears to be started okay in Ambari view, but the it shows stopped in about 3 minutes. FYI. it's a Vanilla HDP-2.4.0.0-169 install plus some configuration changes on disks, HDFS, but I haven't created anything on Hbase. The attached is the screenshot of the HBase view of Ambari. Below are the log at the time of failure: /var/log/hbase/hbase-hbase-master-serverhost.log Could you please advise on where to look at to make it started? Thanks, 2016-05-13 03:56:41,651 INFO [ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 0 tasks={/hbase-unsecure/splitWAL/WALs%2Fip-172-31-25-148.ap-southeast-2.compute.internal%2C16020%2C1463025987708-splitting%2Fip-172-31-25-148.ap-southeast-2.compute.internal%252C16020%252C1463025987708.default.1463030679002=last_update = 1463126191938 last_version = 399 cur_worker_name = ip-172-31-28-128.ap-southeast-2.compute.internal,16020,1463125906252 status = in_progress incarnation = 1 resubmits = 1 batch = installed = 1 done = 0 error = 0} 2016-05-13 03:56:47,651 INFO [ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 0 tasks={/hbase-unsecure/splitWAL/WALs%2Fip-172-31-25-148.ap-southeast-2.compute.internal%2C16020%2C1463025987708-splitting%2Fip-172-31-25-148.ap-southeast-2.compute.internal%252C16020%252C1463025987708.default.1463030679002=last_update = 1463126191938 last_version = 399 cur_worker_name = ip-172-31-28-128.ap-southeast-2.compute.internal,16020,1463125906252 status = in_progress incarnation = 1 resubmits = 1 batch = installed = 1 done = 0 error = 0} 2016-05-13 03:56:53,431 FATAL [ip-172-31-21-148:16000.activeMasterManager] master.HMaster: Failed to become active master java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1005) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:799) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:191) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1783) at java.lang.Thread.run(Thread.java:745) 2016-05-13 03:56:53,433 FATAL [ip-172-31-21-148:16000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: [] 2016-05-13 03:56:53,433 FATAL [ip-172-31-21-148:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown. java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1005) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:799) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:191) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1783) at java.lang.Thread.run(Thread.java:745) 2016-05-13 03:56:53,433 INFO [ip-172-31-21-148:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown. 2016-05-13 03:56:53,433 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] regionserver.HRegionServer: Stopping infoServer 2016-05-13 03:56:53,456 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010 2016-05-13 03:56:53,500 WARN [MASTER_SERVER_OPERATIONS-ip-172-31-21-148:16000-1] master.SplitLogManager: Stopped while waiting for log splits to be completed 2016-05-13 03:56:53,500 WARN [MASTER_SERVER_OPERATIONS-ip-172-31-21-148:16000-1] master.SplitLogManager: error while splitting logs in [hdfs://ip-172-31-23-165.ap-southeast-2.compute.internal:8020/apps/hbase/data/WALs/ip-172-31-25-148.ap-southeast-2.compute.internal,16020,1463025987708-splitting] installed = 1 but only 0 done 2016-05-13 03:56:53,501 ERROR [MASTER_SERVER_OPERATIONS-ip-172-31-21-148:16000-1] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: failed log splitting for ip-172-31-25-148.ap-southeast-2.compute.internal,16020,1463025987708, will retry at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:357) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:220) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://ip-172-31-23-165.ap-southeast-2.compute.internal:8020/apps/hbase/data/WALs/ip-172-31-25-148.ap-southeast-2.compute.internal,16020,1463025987708-splitting] Task = installed = 1 done = 0 error = 0 at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:391) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:364) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:286) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:213) ... 4 more 2016-05-13 03:56:53,502 ERROR [MASTER_SERVER_OPERATIONS-ip-172-31-21-148:16000-3] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: Server is stopped at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:194) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-05-13 03:56:53,556 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] procedure2.ProcedureExecutor: Stopping the procedure executor 2016-05-13 03:56:53,557 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store 2016-05-13 03:56:53,656 INFO [ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935_splitLogManager__ChoreService_1] master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout Monitor was stopped 2016-05-13 03:56:53,691 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] regionserver.HRegionServer: stopping server ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935 2016-05-13 03:56:53,691 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x154a916c5040001 2016-05-13 03:56:53,692 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] zookeeper.ZooKeeper: Session: 0x154a916c5040001 closed 2016-05-13 03:56:53,693 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down 2016-05-13 03:56:53,703 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] regionserver.HRegionServer: stopping server ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935; all regions closed. 2016-05-13 03:56:53,703 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] hbase.ChoreService: Chore service for: ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935 had [[ScheduledChore: Name: ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-ip-172-31-21-148:16000 Period: 300000 Unit: MILLISECONDS]] on shutdown 2016-05-13 03:56:53,706 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x154a916c5040002 2016-05-13 03:56:53,707 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] zookeeper.ZooKeeper: Session: 0x154a916c5040002 closed 2016-05-13 03:56:53,707 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] hbase.ChoreService: Chore service for: ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935_splitLogManager_ had [] on shutdown 2016-05-13 03:56:53,707 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down. 2016-05-13 03:56:53,707 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] ipc.RpcServer: Stopping server on 16000 2016-05-13 03:56:53,707 INFO [ip-172-31-21-148:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down 2016-05-13 03:56:53,716 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/rs/ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935 already deleted, retry=false 2016-05-13 03:56:53,718 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] zookeeper.ZooKeeper: Session: 0x154a916c5040000 closed 2016-05-13 03:56:53,718 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] regionserver.HRegionServer: stopping server ip-172-31-21-148.ap-southeast-2.compute.internal,16000,1463125847935; zookeeper connection closed. 2016-05-13 03:56:53,718 INFO [master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000] regionserver.HRegionServer: master/ip-172-31-21-148.ap-southeast-2.compute.internal/172.31.21.148:16000 exiting 2016-05-13 03:56:53,718 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down

Online	Offline
Last Visited	‎09-12-2016 05:53 AM

Date Registered	‎05-13-2016 07:35 AM
Last Visited	‎09-12-2016 05:53 AM
Total Messages Posted	19
Total Kudos Received	3

Re: Zeppelin installation via Ambari is failing on...

Re: HBase server fails to start - File does not ex...

Re: Sqoop error with Oracle: Could not load db dri...

Re: Sqoop error with Oracle: Could not load db dri...

Re: Sqoop error with Oracle: Could not load db dri...

Re: Sqoop error with Oracle: Could not load db dri...

Re: Sqoop error with Oracle: Could not load db dri...

Re: Sqoop error with Oracle: Could not load db dri...

Sqoop error with Oracle: Could not load db driver ...

Re: HBase server fails to start - File does not ex...

Re: HBase server fails to start - File does not ex...

Re: Zeppelin installation via Ambari is failing on...

Zeppelin installation via Ambari is failing on HDP...

Re: HBase server fails to start - File does not ex...

Re: HBase server fails to start - File does not ex...

Re: HBase server fails to start - File does not ex...

HBase server fails to start - File does not exist:...

Re: HBase master fails to start

Re: HBase master fails to start

HBase master fails to start