question Re: Install and configure new Ambari server for existing HDF cluster in Support Questions

Install and configure new Ambari server for existing HDF cluster

a_gulshani — Tue, 20 Nov 2018 21:29:24 GMT

Hi, all!

Can anyone give any suggestion in configuring new Ambari server with an existing HDF cluster

Previos version of ambari fails in ambari-server start after downgrade

[root@bi-nifi1 dmp]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.............ERROR: Exiting with exit code -1. 
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@bi-nifi1 dmp]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.AmbariException: An error occured during updating current repository versions with stack repositories.
        at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:99)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1061)
Caused by: java.lang.NullPointerException
        at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.updateRepoVersion(UpdateActiveRepoVersionOnStartup.java:106)
        at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:92)
        ... 2 more
Exception in thread "main" org.apache.ambari.server.AmbariException: Error stopping the server
        at org.apache.ambari.server.controller.AmbariServer.stop(AmbariServer.java:880)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1075)
[root@bi-nifi1 dmp]#

Re: Install and configure new Ambari server for existing HDF cluster

jagadeesan — Wed, 21 Nov 2018 03:08:57 GMT

Could you please attach ambari-server.log for more detail about error ?

Re: Install and configure new Ambari server for existing HDF cluster

jagadeesan — Wed, 21 Nov 2018 07:23:49 GMT

@Gulshan Agivetova

This seems to be related to some uncleaned installation. Mostly the "/usr/lib/ambari-server/lib/ambari_commons" and "/usr/lib/python2.6/site-packages/ambari_commons" might have some older version of libraries installation. So please clean up those old python libraries. Added to that setup again ambari-server using ambari-server.py script appropriate to your cluster configuration.

/usr/sbin/ambari-server.py setup --databasehost=localhost --databasename=ambari --databaseusername=ambari --postgresschema=ambari --databasepassword=ambari --databaseport=5432 --database=postgres -s

Re: Install and configure new Ambari server for existing HDF cluster

a_gulshani — Wed, 21 Nov 2018 10:54:12 GMT

Hi @Jagadeesan A S

Thank you for your concern!

How can I identify which libraries are from older versions, there is no version on the file name?

[dmp@bi-nifi1 ~]$ cd /usr/lib/ambari-server/lib/ambari_commons
[dmp@bi-nifi1 ambari_commons]$ ls
aggregate_functions.py     ambari_service.pyc           credential_store_helper.pyo  inet_utils.py     logging_utils.pyc  os_check.pyo        os_utils.py             parallel_processing.pyc  str_utils.pyc        xml_utils.pyo
aggregate_functions.pyc    ambari_service.pyo           exceptions.py                inet_utils.pyc    logging_utils.pyo  os_family_impl.py   os_utils.pyc            parallel_processing.pyo  str_utils.pyo        yaml_utils.py
aggregate_functions.pyo    constants.py                 exceptions.pyc               inet_utils.pyo    network.py         os_family_impl.pyc  os_utils.pyo            resources                urllib_handlers.py   yaml_utils.pyc
ambari_metrics_helper.py   constants.pyc                exceptions.pyo               __init__.py       network.pyc        os_family_impl.pyo  os_windows.py           shell.py                 urllib_handlers.pyc  yaml_utils.pyo
ambari_metrics_helper.pyc  constants.pyo                firewall.py                  __init__.pyc      network.pyo        os_linux.py         os_windows.pyc          shell.pyc                urllib_handlers.pyo
ambari_metrics_helper.pyo  credential_store_helper.py   firewall.pyc                 __init__.pyo      os_check.py        os_linux.pyc        os_windows.pyo          shell.pyo                xml_utils.py
ambari_service.py          credential_store_helper.pyc  firewall.pyo                 logging_utils.py  os_check.pyc       os_linux.pyo        parallel_processing.py  str_utils.py             xml_utils.pyc

Re: Install and configure new Ambari server for existing HDF cluster

a_gulshani — Wed, 21 Nov 2018 10:56:27 GMT

ambari-server.zip

Re: Install and configure new Ambari server for existing HDF cluster

jagadeesan — Wed, 21 Nov 2018 14:14:08 GMT

@Gulshan Agivetova

From the ambari-server log, I can see Ambari could not load version definition for HDP-2.6. I would recommended you to have clean all python libraries, yum cleanup and freshly reinstall. Could you please try below steps. appropriate to your host machine.

Clean Ambari old installation which include cleaning up the old Python libraries installation

yum remove ambari-server ambari-agent -y
rm -f /usr/sbin/ambari*
rm -f /usr/lib/python2.6/site-packages/ambari_commons
rm -rf /usr/lib/python2.6/site-packages/resource_management
rm -rf /usr/lib/python2.6/site-packages/ambari_jinja2
rm -rf /usr/lib/ambari-server
rm -rf /usr/lib/ambari-agent

Get the new repo according to your host machine

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.5.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

Perform a yum cleanup

yum clean all

Now do a fresh install of ambari binaries.

yum install ambari-server -y
yum install ambari-agent -y

Now perform the ambari-server setup again.

/usr/sbin/ambari-server.py setup --databasehost=localhost --databasename=ambari --databaseusername=ambari --postgresschema=ambari --databasepassword=ambari --databaseport=5432 --database=postgres -s

I guess these above steps may resolve your issue.

Re: Install and configure new Ambari server for existing HDF cluster

a_gulshani — Fri, 30 Nov 2018 19:25:27 GMT

Hi @Jagadeesan A S

Thank you for your response!

I have followed all the steps you have stated. After ambari-server setup I dropped the new ambari database, and recovered the database from backup.

And when trying to start ambari-server it gives following error:

ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA

# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start................
DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details.
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@bi-nifi1 backup]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
Database consistency check result: DB_CHECK_WARNING
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA
at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067)
[root@bi-nifi1 backup]#

Is it possible to manually start AMBARI_INFRA service?

Thanks in advance!

Re: Install and configure new Ambari server for existing HDF cluster

jagadeesan — Sat, 01 Dec 2018 18:48:49 GMT

@Gulshan Agivetova

You can force Ambari Server to start by skipping this check with the following option:

ambari-server start --skip-database-check