updateddeleteuser.zipISSUE: Ranger ldap integration was working fine. Customer delete user from ranger UI and was facing issue while re-importing user in Ranger. ROOT CAUSE: Customer removed the users from ranger UI and expected that the user should be automatically imported from ranger usersync process Below are sample screenshot - User named 'testuser' is deleted from Ranger UI. But below you can see the user is still available in database. RESOLUTION: There are multiple tables which has the entry for the user. You need to run delete script to delete the user entries from database and re-start ranger usersync process to re-import the user. Please find attach delete script- Syntax to run the script - $ deleteUser.sh -f input.txt -u ranger_user -p password -db ranger [-r <replaceUser>] ... View more

ISSUE: After enabling Ambari SSL Hive views stopped working. ERROR: 08 Nov 2016 11:32:23,330 WARN [qtp-ambari-client-263] nio:720 - javax.net.ssl.SSLException: Received fatal alert: certificate_unknown 08 Nov 2016 11:32:23,331 ERROR [qtp-ambari-client-256] ServiceFormattedException:100 - org.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambari org.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambari at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:176) at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:142) at org.apache.ambari.view.utils.ambari.AmbariApi.getHostsWithComponent(AmbariApi.java:99) at org.apache.ambari.view.hive.client.ConnectionFactory.getHiveHost(ConnectionFactory.java:79) at org.apache.ambari.view.hive.client.ConnectionFactory.create(ConnectionFactory.java:68) at org.apache.ambari.view.hive.client.UserLocalConnection.initialValue(UserLocalConnection.java:42) at org.apache.ambari.view.hive.client.UserLocalConnection.initialValue(UserLocalConnection.java:26) at org.apache.ambari.view.utils.UserLocal.get(UserLocal.java:66) at org.apache.ambari.view.hive.resources.browser.HiveBrowserService.databases(HiveBrowserService.java:87) at sun.reflect.GeneratedMethodAccessor186.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) Root Cause: Truststore configuration for Ambari Server was missing. Resolution: Setup the trustore for ambari server as per link below after which above issue was resolved. https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Ambari_Security_Guide/content/_set_up_truststore_for_ambari_server.html ... View more

SYMPTOM: During HDP upgrade from 2.3 to 2.5 YARN check is failing due to NoSuchMethodError org.apache.hadoop.yarn.api.records.Resource.getMemorySize()J ERROR: Below was the error in application logs - 16/11/14 10:30:12 FATAL distributedshell.ApplicationMaster: Error running ApplicationMaster java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.records.Resource.getMemorySize()J at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.run(ApplicationMaster.java:585) at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:298) ROOT CAUSE: There was issue with classpath where the nodemanager on which the job was running was pointing to older version[ie. 2.3] classpath. RESOLUTION: There are two solutions as below - 1. Skip this step in Ambari upgrade UI and proceed. Ambari will take care of setting up the classpath. 2. Modify the classpath manually and confirm the set classpath using "hadoop classpath" command and re-run the service check. ... View more

Issue: While performing HDP upgrade it failed on last step. Error Finalizing HDP upgrade Failed on: Save Cluster State Sample screenshot is attached below - Many times error might change, and in my case below was the error - Begin finalizing the upgrade of cluster TEST_DEV to version 2.5.0.0-1245 The following 7 host(s) have not been upgraded to version 2.5.0.0-1245. Please install and upgrade the Stack Version on those hosts and try again. Hosts: host1.example.com, host2.example.com, host3.example.com, host4.example.com, host5.example.com, host6.example.com, host7.example.com Root Cause: When check in ambari DB for "host_version" table we were able to see the 7 hosts were pointing to state as INSTALLED as shown below - The repo_version_id=151 should be in "CURRENT" state which was not the case. It was in INSTALLED state. Solution: 1. First set the previous repo_version_id ie 101 to INSTALLED state using below command - mysql> update host_version set state='INSTALLED' where repo_version_id='101'; 2. Now modify the hosts which was having latest repo_version_id [ie. 151] to CURRENT state - mysql> update host_version set state='CURRENT' where repo_version_id='151'; 3. Restarted ambari server and click on "Retry" on host upgrade screen after which upgrade was successfully completed. ... View more

Issue: Upgrade Ambari server from X.X.X version to Y.Y.Y version which was successful with errors as showing below - sudo ambari-server upgrade Using python /usr/bin/python Upgrading ambari-server Updating properties in ambari.properties ... WARNING: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings Fixing database objects owner Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y Upgrading database schema Error output from schema upgrade command: com.google.inject.ProvisionException: Guice provision errors: 1) null returned by binding at org.apache.ambari.server.state.ServiceComponentHostFactory.createExisting() but parameter 2 of org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl.<init>() is not @Nullable while locating org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity annotated with @com.google.inject.assistedinject.Assisted(value=) for parameter 2 at org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl.<init>(ServiceComponentHostImpl.java:785) while locating org.apache.ambari.server.state.ServiceComponentHost annotated with interface com.google.inject.assistedinject.Assisted 1 error at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987) at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) at com.sun.proxy.$Proxy19.createExisting(Unknown Source) at org.apache.ambari.server.state.ServiceComponentImpl.<init>(ServiceComponentImpl.java:161) at org.apache.ambari.server.state.ServiceComponentImpl$$EnhancerByGuice$$ecf0911c.<init>(<generated>) at org.apache.ambari.server.state.ServiceComponentImpl$$EnhancerByGuice$$ecf0911c$$FastClassByGuice$$5fef485a.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.ProxyFactory$ProxyConstructor.newInstance(ProxyFactory.java:260) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) at com.sun.proxy.$Proxy18.createExisting(Unknown Source) at org.apache.ambari.server.state.ServiceImpl.<init>(ServiceImpl.java:163) at org.apache.ambari.server.state.ServiceImpl$$EnhancerByGuice$$d0875be7.<init>(<generated>) at org.apache.ambari.server.state.ServiceImpl$$EnhancerByGuice$$d0875be7$$FastClassByGuice$$f62c0b29.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.ProxyFactory$ProxyConstructor.newInstance(ProxyFactory.java:260) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) at com.sun.proxy.$Proxy14.createExisting(Unknown Source) at org.apache.ambari.server.state.cluster.ClusterImpl.loadServices(ClusterImpl.java:426) at org.apache.ambari.server.state.cluster.ClusterImpl.getServices(ClusterImpl.java:961) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addNewConfigurationsFromXml(AbstractUpgradeCatalog.java:311) at org.apache.ambari.server.upgrade.UpgradeCatalog212.executeDMLUpdates(UpgradeCatalog212.java:157) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:664) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:228) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:305) com.google.inject.ProvisionException: Guice provision errors: Adjusting ambari-server permissions and ownership... Ambari Server 'upgrade' completed successfully. When tried to start ambari server it got failed with database consistency check failed. Debug: When check ambari database logs we found below error - $cat /var/log/ambari-server/ambari-check-database.log 2016-11-10 09:40:55,365 INFO - ******************************* Check database started ******************************* 2016-11-10 09:40:58,841 INFO - Checking for configs not mapped to any cluster 2016-11-10 09:40:58,858 INFO - Checking for configs selected more than once 2016-11-10 09:40:58,860 INFO - Checking for hosts without state 2016-11-10 09:40:58,861 INFO - Checking host component states count equals host component desired states count 2016-11-10 09:40:58,863 ERROR - Your host component states (hostcomponentstate table) count not equals host component desired states (hostcomponentdesiredstate table) count! 2016-11-10 09:40:58,863 INFO - Checking services and their configs 2016-11-10 09:41:01,195 INFO - ******************************* Check database completed ******************************* Resolution: 1. As per above logs file ie.ambari-check-database.log we see that there is count differences for hostcomponentstate table and hostcomponentdesiredstate table . 2. Tried to do select count(*) on both tables as below and we found 'X' no of rows differences. [X was = 10 in my case] select count(*) on hostcomponentstate; select count(*) on hostcomponentdesiredstate; 3. To find out differences we did "group by" on each column as below - select count(*) from hostcomponentstate group by host_id; select count(*) from hostcomponentdesiredstate group by host_id; Followed same steps for all rest of the columns and was able to find the differences in the count values for the respective columns. 4. We deleted additional entries using delete command as shown below - delete from hostcomponentstate where host_id=**** and version='****'; For Eg. delete from hostcomponentstate where host_id=4 and version='2.3.0.0-2557'; 5. Tried restarting ambari server which worked successfully. 6. Stopped ambari server again as there was error previously while we ran 'ambari-server upgrade' initially. 7. Re-run the ambari-server upgrade command which ran successfully. 8. Started ambari-server, which was able to start without any issue. ... View more