Install and configure new Ambari server for existing HDF cluster

[#1]

Hi, all!

Can anyone give any suggestion in configuring new Ambari server with an existing HDF cluster

Previos version of ambari fails in ambari-server start after downgrade

[root@bi-nifi1 dmp]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.............ERROR: Exiting with exit code -1. 
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@bi-nifi1 dmp]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.AmbariException: An error occured during updating current repository versions with stack repositories.
        at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:99)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1061)
Caused by: java.lang.NullPointerException
        at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.updateRepoVersion(UpdateActiveRepoVersionOnStartup.java:106)
        at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:92)
        ... 2 more
Exception in thread "main" org.apache.ambari.server.AmbariException: Error stopping the server
        at org.apache.ambari.server.controller.AmbariServer.stop(AmbariServer.java:880)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1075)
[root@bi-nifi1 dmp]# 

[#2]

Could you please attach ambari-server.log for more detail about error ?

[#3]

ambari-server.zip

[#4]

@Gulshan Agivetova

This seems to be related to some uncleaned installation. Mostly the "/usr/lib/ambari-server/lib/ambari_commons" and "/usr/lib/python2.6/site-packages/ambari_commons" might have some older version of libraries installation. So please clean up those old python libraries. Added to that setup again ambari-server using ambari-server.py script appropriate to your cluster configuration.

/usr/sbin/ambari-server.py setup --databasehost=localhost --databasename=ambari --databaseusername=ambari --postgresschema=ambari --databasepassword=ambari --databaseport=5432 --database=postgres -s

[#5]

Hi @Jagadeesan A S

Thank you for your concern!

How can I identify which libraries are from older versions, there is no version on the file name?

[dmp@bi-nifi1 ~]$ cd /usr/lib/ambari-server/lib/ambari_commons
[dmp@bi-nifi1 ambari_commons]$ ls
aggregate_functions.py     ambari_service.pyc           credential_store_helper.pyo  inet_utils.py     logging_utils.pyc  os_check.pyo        os_utils.py             parallel_processing.pyc  str_utils.pyc        xml_utils.pyo
aggregate_functions.pyc    ambari_service.pyo           exceptions.py                inet_utils.pyc    logging_utils.pyo  os_family_impl.py   os_utils.pyc            parallel_processing.pyo  str_utils.pyo        yaml_utils.py
aggregate_functions.pyo    constants.py                 exceptions.pyc               inet_utils.pyo    network.py         os_family_impl.pyc  os_utils.pyo            resources                urllib_handlers.py   yaml_utils.pyc
ambari_metrics_helper.py   constants.pyc                exceptions.pyo               __init__.py       network.pyc        os_family_impl.pyo  os_windows.py           shell.py                 urllib_handlers.pyc  yaml_utils.pyo
ambari_metrics_helper.pyc  constants.pyo                firewall.py                  __init__.pyc      network.pyo        os_linux.py         os_windows.pyc          shell.pyc                urllib_handlers.pyo
ambari_metrics_helper.pyo  credential_store_helper.py   firewall.pyc                 __init__.pyo      os_check.py        os_linux.pyc        os_windows.pyo          shell.pyo                xml_utils.py
ambari_service.py          credential_store_helper.pyc  firewall.pyo                 logging_utils.py  os_check.pyc       os_linux.pyo        parallel_processing.py  str_utils.py             xml_utils.pyc

[#6 · Accepted solution]

@Gulshan Agivetova

From the ambari-server log, I can see Ambari could not load version definition for HDP-2.6. I would recommended you to have clean all python libraries, yum cleanup and freshly reinstall. Could you please try below steps. appropriate to your host machine.

• Clean Ambari old installation which include cleaning up the old Python libraries installation

yum remove ambari-server ambari-agent -y
rm -f /usr/sbin/ambari*
rm -f /usr/lib/python2.6/site-packages/ambari_commons
rm -rf /usr/lib/python2.6/site-packages/resource_management
rm -rf /usr/lib/python2.6/site-packages/ambari_jinja2
rm -rf /usr/lib/ambari-server
rm -rf /usr/lib/ambari-agent

• Get the new repo according to your host machine

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.5.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

• Perform a yum cleanup

yum clean all

• Now do a fresh install of ambari binaries.

yum install ambari-server -y
yum install ambari-agent -y

• Now perform the ambari-server setup again.

/usr/sbin/ambari-server.py setup --databasehost=localhost --databasename=ambari --databaseusername=ambari --postgresschema=ambari --databasepassword=ambari --databaseport=5432 --database=postgres -s

I guess these above steps may resolve your issue.

[#7]

Hi @Jagadeesan A S

Thank you for your response!

I have followed all the steps you have stated. After ambari-server setup I dropped the new ambari database, and recovered the database from backup.

And when trying to start ambari-server it gives following error:

ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA

# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start................
DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details.
ERROR: Exiting with exit code -1. 
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@bi-nifi1 backup]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
Database consistency check result: DB_CHECK_WARNING
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA
        at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
        at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067)
[root@bi-nifi1 backup]# 

Is it possible to manually start AMBARI_INFRA service?

Thanks in advance!

[#8]

@Gulshan Agivetova

You can force Ambari Server to start by skipping this check with the following option:

ambari-server start --skip-database-check