After completing Ambari upgrade from Ambari 1.7 to Ambari 2.0.2, Ambari failed to start because of below error(s):
[root@xyz ~]# service ambari-server start
Using python /usr/bin/python2.6
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process died with exitcode 255. Check /var/log/ambari-server/ambari-server.out for more information.
.
If you check /var/log/ambari-server/ambari-server.out logs, you get below error(s):
[root@xyz ~]# cat /var/log/ambari-server/ambari-server.out
[EL Warning]: metadata: 2016-12-01 10:01:23.436--ServerSession(306824177)--The reference column name [resource_type_id] mapped on the element [field permissions] does not correspond to a valid id or basic field/column on the mapping reference. Will use referenced column name as provided.
[EL Info]: 2016-12-01 10:01:24.412--ServerSession(306824177)--EclipseLink, version: Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd
[EL Info]: connection: 2016-12-01 10:01:24.603--ServerSession(306824177)--file:/usr/lib/ambari-server/ambari-server-2.0.2.25.jar_ambari-server_url=jdbc:postgresql://localhost/ambari_user=ambari login successful
.
Note - Above error is complex and confusing. You should always check /var/log/ambari-server/ambari-server.log.
In my case, I was getting below error in ambari-server.log
org.apache.ambari.server.StackAccessException: Stack data, stackName=HDP, stackVersion=2.2, serviceName=SMARTSENSE
at org.apache.ambari.server.api.services.AmbariMetaInfo.getService(AmbariMetaInfo.java:497)
at org.apache.ambari.server.api.services.AmbariMetaInfo.getComponent(AmbariMetaInfo.java:265)
at org.apache.ambari.server.controller.utilities.DatabaseChecker.checkDBConsistency(DatabaseChecker.java:96)
at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:217)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:665)
.
Why this has happened?
This is one of the Known issue - Sometimes stack component for SmartSense gets removed from Ambari DB after an upgrade.
.
How to resolve this?
1. Check what version of SmartSense you have installed on your cluster.
2. Download RPM from Customer Support portal
3. Run below command to re-add the SmartSense view and Stack to the Ambari DB.
[root@xyz crazyadmins.com]# hst add-to-ambari
Enter SmartSense distributable path: /home/kuldeepk/smartsense-hst-1.2.1.0-161.x86_64.rpm
Added SmartSense service definition to Ambari
4. Restart 'ambari-server'.
[root@xyz crazyadmins.com]# ambari-server restart
Using python /usr/bin/python
Restarting ambari-server
Using python /usr/bin/python
Stopping ambari-server
Ambari Server is not running
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................
Ambari Server 'start' completed successfully.
[root@prodnode1 ~]#
.
Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! 🙂