Ok, I seem to have hive up and running with mysql, but now I cannot get ranger admin to start: stderr: /var/lib/ambari-agent/data/errors-1023.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 155, in <module> RangerAdmin().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 512, in restart self.stop(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 54, in stop Execute(format('{params.ranger_stop}'), environment={'JAVA_HOME': params.java_home}, user=params.unix_user) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/usr/bin/ranger-admin-stop' returned 126. -bash: /usr/bin/ranger-admin-stop: Permission denied ============================= stdout: /var/lib/ambari-agent/data/output-1023.txt 2016-10-25 10:58:39,550 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.2.0-258 2016-10-25 10:58:39,550 - Checking if need to create versioned conf dir /etc/hadoop/2.4.2.0-258/0 2016-10-25 10:58:39,550 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.2.0-258 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-10-25 10:58:39,587 - call returned (1, '/etc/hadoop/2.4.2.0-258/0 exist already', '') 2016-10-25 10:58:39,588 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.2.0-258 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-10-25 10:58:39,616 - checked_call returned (0, '') 2016-10-25 10:58:39,616 - Ensuring that hadoop has the correct symlink structure 2016-10-25 10:58:39,617 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-10-25 10:58:39,736 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.2.0-258 2016-10-25 10:58:39,736 - Checking if need to create versioned conf dir /etc/hadoop/2.4.2.0-258/0 2016-10-25 10:58:39,737 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.2.0-258 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-10-25 10:58:39,763 - call returned (1, '/etc/hadoop/2.4.2.0-258/0 exist already', '') 2016-10-25 10:58:39,764 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.2.0-258 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-10-25 10:58:39,791 - checked_call returned (0, '') 2016-10-25 10:58:39,792 - Ensuring that hadoop has the correct symlink structure 2016-10-25 10:58:39,792 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-10-25 10:58:39,794 - Group['kms'] {} 2016-10-25 10:58:39,795 - Group['spark'] {} 2016-10-25 10:58:39,796 - Group['ranger'] {} 2016-10-25 10:58:39,797 - Group['hadoop'] {} 2016-10-25 10:58:39,797 - Group['users'] {} 2016-10-25 10:58:39,800 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,801 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,802 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,803 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger']} 2016-10-25 10:58:39,804 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2016-10-25 10:58:39,804 - User['kms'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,805 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,806 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2016-10-25 10:58:39,807 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,808 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,809 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,809 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,810 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2016-10-25 10:58:39,811 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-10-25 10:58:39,814 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2016-10-25 10:58:39,819 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2016-10-25 10:58:39,820 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'} 2016-10-25 10:58:39,821 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-10-25 10:58:39,822 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2016-10-25 10:58:39,827 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if 2016-10-25 10:58:39,828 - Group['hdfs'] {} 2016-10-25 10:58:39,828 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']} 2016-10-25 10:58:39,829 - FS Type: 2016-10-25 10:58:39,829 - Directory['/etc/hadoop'] {'mode': 0755} 2016-10-25 10:58:39,847 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2016-10-25 10:58:39,848 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777} 2016-10-25 10:58:39,863 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2016-10-25 10:58:39,881 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-10-25 10:58:39,883 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'} 2016-10-25 10:58:39,883 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'} 2016-10-25 10:58:39,889 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'} 2016-10-25 10:58:39,891 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'} 2016-10-25 10:58:39,892 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2016-10-25 10:58:39,909 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'} 2016-10-25 10:58:39,910 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2016-10-25 10:58:39,911 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} 2016-10-25 10:58:39,917 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'} 2016-10-25 10:58:39,922 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} 2016-10-25 10:58:40,092 - Execute['/usr/bin/ranger-admin-stop'] {'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.8.0_60'}, 'user': 'ranger'} I created the database and user accounts in mysql it needs, but I don't see why it would give a "permission denied" error attempting to stop ranger-admin. I tried running the script manually from the root account, to no avail. ... View more

I ran the setup script, which apparently was ok. ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Using python /usr/bin/python Setup ambari-server Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully. mysql has been set up with a hive account using the recommended approach: # mysql -u root -p CREATE USER ‘<HIVEUSER>’@’localhost’ IDENTIFIED BY ‘<HIVEPASSWORD>’; GRANT ALL PRIVILEGES ON *.* TO '<HIVEUSER>'@'localhost'; CREATE USER ‘<HIVEUSER>’@’%’ IDENTIFIED BY ‘<HIVEPASSWORD>’; GRANT ALL PRIVILEGES ON *.* TO '<HIVEUSER>'@'%'; CREATE USER '<HIVEUSER>'@'<HIVEMETASTOREFQDN>'IDENTIFIED BY '<HIVEPASSWORD>'; GRANT ALL PRIVILEGES ON *.* TO '<HIVEUSER>'@'<HIVEMETASTOREFQDN>'; FLUSH PRIVILEGES; But when I go to add the hive service from ambari (I set the metastore to be on the secondary namenode) and test the connection: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 477, in <module> CheckHost().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 212, in actionexecute raise Fail(error_message) resource_management.core.exceptions.Fail: Check db_connection_check was unsuccessful. Exit code: 1. Message: Error: Ambari Server cannot download the database JDBC driver and is unable to test the database connection. You must run ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/your/mysql/driver.jar on the Ambari Server host to make the JDBC driver available for download and to enable testing the database connection. HTTP Error 404: Not Found stdout: DB connection check started. WARNING: File /var/lib/ambari-agent/cache/DBConnectionVerification.jar already exists, assuming it was downloaded before Error: Ambari Server cannot download the database JDBC driver and is unable to test the database connection. You must run ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/your/mysql/driver.jar on the Ambari Server host to make the JDBC driver available for download and to enable testing the database connection. HTTP Error 404: Not Found I entered in the correct password at the prompt for the hive account I created on mysql, but I keep getting this error. What step could I be missing? ... View more

We did some preliminary tests and it seems distcp with -strategy dynamic improves performance by a substation amount on our workload. Digging through what documentation I can find, it does say that it improves performance with MOST workloads, but I can't find any clear guidance on what workloads it would perform poorly with. 1. If it is so much better in most situations, why isn't -strategy dynamic the Hadoop default? 2. What are the potential downsides to using it by default? Is there any use case where -strategy uniform would perform better? ... View more

I would like to export HBase snapshot data to a backup appliance outside of hdfs for disaster recovery. I can use ExportSnapshot to export the table to a non hdfs URI, no problem. But there is no ImportSnapshot - if I try to use ExportSnapshot to bring it back from the external source outside the cluster, using --copy-from <Backup-appliance-URI>/hbase and --copy-to /hbase complains about an unexpected (non hdfs) URI. If I use DistCp to copy it to a staging area on the hdfs cluster first then use ExportSnapshot to bring it back it works but is undesirable because I need to create a temporary staging area. I found I was able to use DistCp directly on the exported snapshot and copy it back to hdfs://hbase and restore the table. This looks like a good solution and seems to work, but are there any unexpected problems I can run into? Is this a recommended approach? ... View more