Problem Statement : The hdfs encryption guide for HDP 3.0.1 states "In Ambari, replace the current value of dfs.permissions.superusergroup with the group name “operator”. however, this option is not available from Ambari. RootCause : As per fix of Apache bug : https://issues.apache.org/jira/browse/AMBARI-22086 ,ambari considers the dfs.permissions.superusergroup as a group property and by default the group property is not meant to be edited in ambari-ui. So the ui disabled the editing of this property by default. Workaround : Navigate to ambari-server and change this config via configs.py /var/lib/ambari-server/resources/scripts/configs.py -l <AMBARI_HOSTNAME>-t 8080-u <USER_NAME>-p <PASSWORD>-a <GET/SET/DELETE>-n <CLUSTER_NAME>-c <CONFIG_TYPE>-k <KEY>-v <VALUE> For ex : [root@asnaik-asnaik1 ~]# /var/lib/ambari-server/resources/scripts/configs.py --help Usage: configs.py [options] Options: -h, --help show this help message and exit -t PORT, --port=PORT Optional port number for Ambari server. Default is '8080'. Provide empty string to not use port. -s PROTOCOL, --protocol=PROTOCOL Optional support of SSL. Default protocol is 'http' -a ACTION, --action=ACTION Script action: <get>, <set>, <delete> -l HOST, --host=HOST Server external host name -n CLUSTER, --cluster=CLUSTER Name given to cluster. Ex: 'c1' -c CONFIG_TYPE, --config-type=CONFIG_TYPE One of the various configuration types in Ambari. Ex: core-site, hdfs-site, mapred-queue-acls, etc. To specify credentials please use "-e" OR "-u" and "-p'": -u USER, --user=USER Optional user ID to use for authentication. Default is 'admin' -p PASSWORD, --password=PASSWORD Optional password to use for authentication. Default is 'admin' -e CREDENTIALS_FILE, --credentials-file=CREDENTIALS_FILE Optional file with user credentials separated by new line. To specify property(s) please use "-f" OR "-k" and "-v'": -f FILE, --file=FILE File where entire configurations are saved to, or read from. Supported extensions (.xml, .json>) [root@asnaik-asnaik1 ~]# /var/lib/ambari-server/resources/scripts/configs.py -l asnaik1 -t 8080 -u admin -p admin -a set -n asnaik -c hdfs-site -k dfs.permissions.superusergroup -v hdfs,operator 2018-12-10 15:19:00,604 INFO ### Performing "set": 2018-12-10 15:19:00,604 INFO ### new property - "dfs.permissions.superusergroup":"hdfs,operator" 2018-12-10 15:19:00,663 INFO ### on (Site:hdfs-site, Tag:version1543379050314) 2018-12-10 15:19:00,675 INFO ### PUTting json into: doSet_version1544455140675467.json 2018-12-10 15:19:00,767 INFO ### NEW Site:hdfs-site, Tag:version1544455140675467 [root@asnaik-asnaik1 ~]# /var/lib/ambari-server/resources/scripts/configs.py -l asnaik1 -t 8080 -u admin -p admin -a get -n asnaik -c hdfs-site -k dfs.permissions.superusergroup |grep -i dfs.permissions.superusergroup ... View more

Problem statement : i am upgrading my ambari-2.6.2.2 to ambari-2.7.3 and the ambari-server upgrade commands fails with exception : java.sql.SQLSyntaxErrorException: Unknown table 'ambari_configuration' in information_schema [root@slambe-1 java]# ambari-server upgrade Using python /usr/bin/python Upgrading ambari-server INFO: Upgrade Ambari Server INFO: Updating Ambari Server properties in ambari.properties ... INFO: Updating Ambari Server properties in ambari-env.sh ... INFO: Original file ambari-env.sh kept INFO: Fixing database objects owner Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y INFO: Upgrading database schema ERROR: Unexpected ValueError: No JSON object could be decoded For more info run ambari-server with -v or --verbose option when i see ambari-server logs i see the following exception: 2018-12-28 13:59:07,062 INFO [main] DBAccessorImpl:869 - Executing query: CREATE TABLE ambari_configuration (category_name VARCHAR(100) NOT NULL, property_name VARCHAR(100) NOT NULL, property_value VARCHAR(2048)) ENGINE=INNODB 2018-12-28 13:59:07,087 ERROR [main] SchemaUpgradeHelper:209 - Upgrade failed. java.sql.SQLSyntaxErrorException: Unknown table 'ambari_configuration' in information_schema at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1218) at com.mysql.cj.jdbc.DatabaseMetaData$7.forEach(DatabaseMetaData.java:2965) at com.mysql.cj.jdbc.DatabaseMetaData$7.forEach(DatabaseMetaData.java:2953) at com.mysql.cj.jdbc.IterateBlock.doForAll(IterateBlock.java:56) at com.mysql.cj.jdbc.DatabaseMetaData.getPrimaryKeys(DatabaseMetaData.java:3006) at org.apache.ambari.server.orm.DBAccessorImpl.tableHasPrimaryKey(DBAccessorImpl.java:1086) at org.apache.ambari.server.orm.DBAccessorImpl.addPKConstraint(DBAccessorImpl.java:577) at org.apache.ambari.server.orm.DBAccessorImpl.addPKConstraint(DBAccessorImpl.java:588) at org.apache.ambari.server.upgrade.UpgradeCatalog270.addAmbariConfigurationTable(UpgradeCatalog270.java:989) at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDDLUpdates(UpgradeCatalog270.java:319) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:970) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:450) 2018-12-28 13:59:07,093 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed org.apache.ambari.server.AmbariException: Unknown table 'ambari_configuration' in information_schema at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:210) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:450) Caused by: java.sql.SQLSyntaxErrorException: Unknown table 'ambari_configuration' in information_schema at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) Root Cause : If you are facing the abouve exception while upgrading to ambari-2.7.0 versions the main reason will be mysql-connector.jar . Please upgrade to supported MySQL connector jar ,currently mysql-5.7 is supported version for ambari, please download connector that supports this version Please download the MYSQL Connector jar from url : https://dev.mysql.com/downloads/connector/j/5.1.html Download MYSQL connector 5.1.47 .jar from this url Change the the mysql-connector using the following docs : https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/administering-ambari/content/amb_using_hive_with_mysql.html and retry the ambari-server upgrade operation again. Please contact Hortonworks support if its a production cluster and you cannot do the steps of your own ... View more

Problem Statement : when using a local repository, installation fails due to empty baseurls being written to the Ambari repository (even though local repository baseurl values were provided). The installation fails with below error : stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 37, in <module> BeforeInstallHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 33, in hook install_packages() File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/shared_initialization.py", line 37, in install_packages retry_count=params.agent_stack_retry_count) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install self._pkg_manager.install_package(package_name, self.__create_context()) File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package shell.repository_manager_executor(cmd, self.properties, context) File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 753, in repository_manager_executor raise RuntimeError(message) RuntimeError: Failed to execute command '/usr/bin/yum -y install hdp-select', exited with code '1', message: 'Repository InstallMedia is listed more than once in the configuration One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid> or subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands,so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: HDP-3.1-repo-1 ' Command aborted. Reason: 'Server considered task failed and automatically aborted it' stdout: We can See that Ambari UI shows the empty baseURL in review step : It also sends the empty Repo version while deploy stage Root cause : https://issues.apache.org/jira/browse/AMBARI-25069 Workaround : This is a Javascript bug in ambari that happens when using local repository and there is no internet access to cluster to workaround this Steps 1) go to /usr/lib/ambari-server/web/javascipts cd /usr/lib/ambari-server/web/javascripts 2) take backup of app.js cp app.js app.js_backup 3) edit the app.js find out the line(39892) : onNetworkIssuesExist: function () { Change the line from : /** * Use Local Repo if some network issues exist */ onNetworkIssuesExist: function () { if (this.get('networkIssuesExist')) { this.get('content.stacks').forEach(function (stack) { stack.setProperties({ usePublicRepo: false, useLocalRepo: true }); stack.cleanReposBaseUrls(); }); } }.observes('networkIssuesExist'), to /** * Use Local Repo if some network issues exist */ onNetworkIssuesExist: function () { if (this.get('networkIssuesExist')) { this.get('content.stacks').forEach(function (stack) { if(stack.get('useLocalRepo') != true){ stack.setProperties({ usePublicRepo: false, useLocalRepo: true }); stack.cleanReposBaseUrls(); } }); } }.observes('networkIssuesExist'), as per : https://github.com/apache/ambari/pull/2743/files Later as you have already deployed the cluster we need to reset the cluster (Caution : this will erase all the configs you have created previously in Step6 and also the Hosts and services you have selected need to select again ) Command : ambari-server reset And hard reload the page and start the create cluster wizard again.   Incase you have already at Step 9 and cannot proceed with ambari-server reset (as it invovles lots of Configs being added again , the below steps are for you ) Preqrequesties : The cluster now is in Deployment step(step 9 ) and you have only retry button to press steps 1) Stop ambari-server 2) login to Database 3) use the below command to list out all the contents in repo_definition table : select * from repo_definition; 4) you can see the base_url will be empty for the all the Rows in the table 5) Correct the base_url for every rows and update it using the command : update repo_definition set base_url='<YOUR BASE URL>' where id=<THE CORESPONDING ID>; for ex : update repo_definition set base_url='http://asnaik.example.com/localrepo/HDP-3.1' where id=9; 6) after correcting all the base_url columns in repo_definition table and also delete the empty repos created by ambari from location /etc/yum.repos.d 7) start ambari, Login to UI and press retry button, The Installation will work as smooth as it can be. Hope this helps. ... View more