can't stop ambari server

mivkovic · ‎11-13-2018

Hi,

I try to stop ambari server and the command runs fine, tells me it stopped. But I can still login to UI and I still see new processes created at the time I issued the command, 21:38.

# ambari-server stop
Using python  /usr/bin/python
Stopping ambari-server
Waiting for server stop...
Ambari Server stopped

# ps -ef|grep ambar
tladmin  23536     1 41 21:04 ?        00:14:11 /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer
postgres 23563  1969  0 21:04 ?        00:00:00 postgres: ambari ambari 127.0.0.1(45338) idle
postgres 23565  1969  0 21:04 ?        00:00:02 postgres: ambari ambari 127.0.0.1(45350) idle
postgres 23662  1969  0 21:05 ?        00:00:02 postgres: ambari ambari 127.0.0.1(45412) idle
postgres 23674  1969  0 21:05 ?        00:00:00 postgres: ambari ambari 127.0.0.1(45414) idle
postgres 23829  1969  0 21:07 ?        00:00:00 postgres: ambari ambari 127.0.0.1(45544) idle
postgres 23830  1969  0 21:07 ?        00:00:00 postgres: ambari ambari 127.0.0.1(45546) idle
postgres 23831  1969  0 21:07 ?        00:00:00 postgres: ambari ambari 127.0.0.1(45548) idle
postgres 24382  1969  0 21:10 ?        00:00:02 postgres: ambari ambari 127.0.0.1(45702) idle
root     29277     1  0 21:38 ?        00:00:00 bash /etc/init.d/ambari-server start
root     29293 29277 25 21:38 ?        00:00:01 /usr/bin/python /usr/sbin/ambari-server.py start
root     29357 29293  0 21:38 ?        00:00:00 /bin/sh -c ulimit -n 10000 ; /bin/su tladmin -s /bin/sh -c '. /var/lib/ambari-server/ambari-env.sh && /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true   -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar' org.apache.ambari.server.controller.AmbariServer > /var/log/ambari-server/ambari-server.out 2>&1 || echo $? > /var/run/ambari-server/ambari-server.exitcode'
root     29358 29357  0 21:38 ?        00:00:00 /bin/su tladmin -s /bin/sh -c . /var/lib/ambari-server/ambari-env.sh && /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true   -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer > /var/log/ambari-server/ambari-server.out 2>&1 || echo $? > /var/run/ambari-server/ambari-server.exitcode
tladmin  29359 29358  0 21:38 ?        00:00:00 sh -c . /var/lib/ambari-server/ambari-env.sh && /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true   -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer > /var/log/ambari-server/ambari-server.out 2>&1 || echo $? > /var/run/ambari-server/ambari-server.exitcode
tladmin  29360 29359 99 21:38 ?        00:00:04 /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer

Auto-start services is disabled. This is Ambari 2.5.2.

Why are there so many AmbariServer processes? If I login on another ambari server (same setup/versions, just different cluster), I see only one AmbariServer process.

Thanks

jsensharma · ‎11-13-2018

@Marjana Ivkovic

The problem seems to be happening because one of the process seems to be started as "tladmin" and the other one as "root"

Example1:

tladmin  23536     1 41 21:04 ?        00:14:11 /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:.........

Example2:

root     29277     1  0 21:38 ?        00:00:00 bash /etc/init.d/ambari-server start
root     29293 29277 25 21:38 ?        00:00:01 /usr/bin/python /usr/sbin/ambari-server.py start
root     29357 29293  0 21:38 ?        00:00:00 /bin/sh -c ulimit -n 10000 ; /bin/su tladmin -s ......

.

I guess it is better to kill the process (PID) and then freshly start the ambari server.

Definitely you might also be seeing the Port 8080 bind exception in your ambari server log:

# grep -i 'BindException' /var/log/ambari-server/ambari-server.log

mivkovic · ‎11-14-2018

I have tried killing all of the processes but they get created right away. I do start ambari-server as root and it is supposed to use tladmin user to start it, as it does properly. I used kill command to stop all processes at 23:30 and didn't issue start, but, as you can see below, they get created somehow.

[root@prodqa-ambari01 ~]# ps -ef|grep ambari-ser
root      6554     1  0 23:31 ?        00:00:00 bash /etc/init.d/ambari-server start
root      6571  6554  0 23:31 ?        00:00:01 /usr/bin/python /usr/sbin/ambari-server.py start
root      6635  6571  0 23:31 ?        00:00:00 /bin/sh -c ulimit -n 10000 ; /bin/su tladmin -s /bin/sh -c '. /var/lib/ambari-server/ambari-env.sh && /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true   -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar' org.apache.ambari.server.controller.AmbariServer > /var/log/ambari-server/ambari-server.out 2>&1 || echo $? > /var/run/ambari-server/ambari-server.exitcode'
root      6636  6635  0 23:31 ?        00:00:00 /bin/su tladmin -s /bin/sh -c . /var/lib/ambari-server/ambari-env.sh && /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true   -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer > /var/log/ambari-server/ambari-server.out 2>&1 || echo $? > /var/run/ambari-server/ambari-server.exitcode
tladmin   6637  6636  0 23:31 ?        00:00:00 sh -c . /var/lib/ambari-server/ambari-env.sh && /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true   -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer > /var/log/ambari-server/ambari-server.out 2>&1 || echo $? > /var/run/ambari-server/ambari-server.exitcode
tladmin   6638  6637 10 23:31 ?        00:01:59 /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer

jsensharma · ‎11-14-2018

@Marjana Ivkovic

Ambari Server might be configured to run as a service. So as soon as you will be killing it the OS might be restarting the service back.

Can you please check if your Ambari Server AutoStart is enabled by any chance as mentioned in the SupportKB? If yes then you can try disabling it or reboot the host once.

https://community.hortonworks.com/content/supportkb/151076/how-to-enable-ambari-server-auto-start-on...

mivkovic · ‎11-14-2018

Our sysadmin helped me with this. It was initctl that was not set correctly (it was respawning). It now shows:

# initctl status ambari-server
ambari-server stop/waiting

And before it was showing:

# initctl status ambari-server
ambari-server start/running, process 13674<br>