Support Questions
Find answers, ask questions, and share your expertise

Re: ambari-agent connection refused after host reboot

@Geoffrey Shelton Okot


Already done /etc/hosts modifications and with this result:

bigdata@bigdata:~$ cat /etc/hosts 
IP                FQDN                  ALIAS 
------------------------------------------------
10.61.2.10       bigdatapruebas.es      bigdata.es 
#10.61.2.10             bigdatapruebas.es #bigdata.es bigdata bigdata.pruebasenergia.es 

::1                     localhost ip6-localhost ip6-loopback 
ff02::1                 ip6-allnodes 
ff02::02                ip6-allrouters

bigdata@bigdata:~$ hotname -f No se ha encontrado la orden «hotname», quizás quiso decir:
  La orden «hostname» del paquete «hostname» (main) hotname: no se encontró la orden 
bigdata@bigdata:~$ hostname -f
bigdatapruebas.es 
bigdata@bigdata:~$


But happens the same thing. When applying changes it works but not working if rebooting until I manually restart agent 😞

Re: ambari-agent connection refused after host reboot

New Contributor

It is not recommended to update ambari DB directly but as in this case the cmd 'ambari-server update-host-names host_names_changes.json' is not a much help, we can perform below actions by taking ambari DB back up. I tested and it worked.

 

Note: Please take DB backup! and PLEASE only do at your own risk.

 

In this case, the new host_name is ambarinn.cluster.com and new name is ambarinn261.cluster.com

We need to concentrate on the host_name that says healthStatus":"UNKNOWN".

 

Steps to resolve:

ambari-agent and ambari-server successfully stopped


[root@ambarinn261 ~]# su - postgres
Last login: Wed Jan 27 13:14:40 EST 2021 on pts/0
-bash-4.2$ psql
psql (9.2.24)
Type "help" for help.

postgres=# \c ambari
You are now connected to database "ambari" as user "postgres".
ambari=# select host_id,host_name,discovery_status,last_registration_time,public_host_name from ambari.hosts;
host_id | host_name | discovery_status | last_registration_time | public_host_name
---------+-------------------------+------------------+------------------------+-------------------------
201 | ambarinn261.cluster.com | | 1611772466200 | ambarinn261.cluster.com
1 | ambarinn.cluster.com | | 1611768543457 | ambarinn.cluster.com
(2 rows)

ambari=# select * from ambari.hoststate;
agent_version | available_mem | current_state | health_status | host_id | time_in_state | maintenance_state
-----------------------+---------------+---------------+----------------------------------------------+---------+---------------+-------------------
{"version":"2.6.1.5"} | 279948 | INIT | {"healthStatus":"HEALTHY","healthReport":""} | 201 | 1611772466200 |
{"version":"2.6.1.5"} | 2120176 | INIT | {"healthStatus":"UNKNOWN","healthReport":""} | 1 | 1611768543457 |
(2 rows)

ambari=# UPDATE ambari.hosts SET host_name='ambarinn261.cluster.com' WHERE host_id=1;
ERROR: duplicate key value violates unique constraint "uq_hosts_host_name"
DETAIL: Key (host_name)=(ambarinn261.cluster.com) already exists.
ambari=# UPDATE ambari.hosts SET public_host_name='ambarinn261.cluster.com' WHERE host_id=1;
UPDATE 1
ambari=# UPDATE ambari.hosts SET public_host_name='ambarinn261a.cluster.com' WHERE host_id=201;
UPDATE 1
ambari=# UPDATE ambari.hosts SET host_name='ambarinn261a.cluster.com' WHERE host_id=201;
UPDATE 1
ambari=# UPDATE ambari.hosts SET host_name='ambarinn261.cluster.com' WHERE host_id=1;
UPDATE 1
ambari=# \q
-bash-4.2$ exit
logout
[root@ambarinn261 ~]# ambari-server start

 

 

After ambari DB is fixed.png

 

Re: ambari-agent connection refused after host reboot

Mentor

@Adrián Gil

Let's discard the ALIAS what I suggest is completely remove it and have your /etc/hosts look like this

10.61.2.10       bigdatapruebas.es      
 ::1                     localhost ip6-localhost ip6-loopback 
ff02::1                 ip6-allnodes 
ff02::02                ip6-allrouters


Stop the ambari-server and agent

# ambari-server stop
# ambari-agent stop


Create a file host_names_changes.json file with hostnames changes.

Contents of the host_names_changes.json

# Ambari host name change


{
  "bigdata" : {
      "old_name_here" : "bigdatapruebas.es"
  }
}

Update the host name for the various components

# Hive

grant all privileges on hive.* to 'hive'@'bigdatapruebas.es' identified by 'hive_password';
grant all privileges on hive.* to 'hive'@'bigdatapruebas.es' with grant option;

# Oozie

grant all privileges on oozie.* to 'hive'@'bigdatapruebas.es' identified by 'oozie_password';
grant all privileges on oozie.* to 'hive'@'bigdatapruebas.es' with grant option;

# Ranger

grant all privileges on ranger.* to 'hive'@'bigdatapruebas.es' identified by 'ranger_password';
grant all privileges on ranger.* to 'hive'@'bigdatapruebas.es' with grant option;

# Rangerkms

grant all privileges on rangerkms.* to 'hive'@'bigdatapruebas.es' identified by 'rangerkms_password';
grant all privileges on rangerkms.* to 'hive'@'bigdatapruebas.es' with grant option;

Change these 3 values in /etc/ambari-server/conf/ambari.properties to the new hostname

server.jdbc.rca.url=
server.jdbc.url=jdbc=
server.jdbc.hostname=

Edit the ambari-agent.ini under server replace the hostname with new Ambari hostname

[server]
hostname=bigdatapruebas.es
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30

Start the ambari-agent

# ambari-agent start

Start the Ambari

# ambari-server start

Open the web browser

http://bigdatapruebas.es:8080

Start all components

Please revert

Re: ambari-agent connection refused after host reboot

@Geoffrey Shelton Okot

I did it and I think it's all right, I show terminal output. At the beginning I thougt it was all right but then heartbeat lost after rebooting again

bigdata@bigdata:~$ hostname -f
bigdata.pruebasenergia.es
bigdata@bigdata:~$ sudo cat /etc/hosts

10.61.2.10              bigdata.pruebasenergia.es
::1                     localhost ip6-localhost ip6-loopback
ff02::1                 ip6-allnodes
ff02::02                ip6-allrouters

bigdata@bigdata:~$ sudo ambari-server stop
Using python  /usr/bin/python
Stopping ambari-server
Ambari Server is not running
bigdata@bigdata:~$
bigdata@bigdata:~$
bigdata@bigdata:~$ sudo ambari-agent stop
Verifying Python version compatibility...
Using python  /usr/bin/python
Found ambari-agent PID: 1601
Stopping ambari-agent
Removing PID file at /run/ambari-agent/ambari-agent.pid
ambari-agent successfully stopped
bigdata@bigdata:~$ sudo vi host_names_changes.json
{
  "bigdata" : {
      "bigdatapruebas.es" : "bigdata.pruebasenergia.es"
  }
}
bigdata@bigdata:~$
bigdata@bigdata:~$ sudo vi /etc/ambari-server/conf/ambari.properties
bigdata@bigdata:~$ cat /etc/ambari-server/conf/ambari.properties | grep server.jdbc
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.driver=org.postgresql.Driver
server.jdbc.driver.path=/usr/ambari/postgresql.jar
server.jdbc.hostname=bigdata.pruebasenergia.es
server.jdbc.port=5432
server.jdbc.postgres.schema=ambari
server.jdbc.rca.driver=org.postgresql.Driver
server.jdbc.rca.url=jdbc:postgresql://bigdata.pruebasenergia.es:5432/ambari
server.jdbc.rca.user.name=ambari
server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat
server.jdbc.url=jdbc:postgresql://bigdata.pruebasenergia.es:5432/ambari
server.jdbc.user.name=ambari
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat
bigdata@bigdata:~$
bigdata@bigdata:~$
bigdata@bigdata:~$ sudo vi /etc/ambari-agent/conf/ambari-agent.ini
bigdata@bigdata:~$ cat /etc/ambari-agent/conf/ambari-agent.ini | grep hostname
hostname = bigdata.pruebasenergia.es
bigdata@bigdata:~$
bigdata@bigdata:~$
bigdata@bigdata:~$ sudo ambari-server update-host-names host_names_changes.json
Using python  /usr/bin/python
Updating host names
Please, confirm Ambari services are stopped [y/n] (n)? y
Please, confirm there are no pending commands on cluster [y/n] (n)? y
Please, confirm you have made backup of the Ambari db [y/n] (n)? y
Ambari Server 'update-host-names' completed successfully.
bigdata@bigdata:~$
bigdata@bigdata:~$
bigdata@bigdata:~$ sudo ambari-agent start
Verifying Python version compatibility...
Using python  /usr/bin/python
Checking for previously running Ambari Agent...
ERROR: ambari-agent already running
Check /run/ambari-agent/ambari-agent.pid for PID.
bigdata@bigdata:~$
bigdata@bigdata:~$ sudo ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.........
............
Server started listening on 8080
DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details.
Ambari Server 'start' completed successfully.
bigdata@bigdata:~$

In databse I've all granted.

Re: ambari-agent connection refused after host reboot

Mentor

@Adrián Gil

Ping me on LinkedIn I could help you remotely

Re: ambari-agent connection refused after host reboot

Mentor

@Adrián Gil

Can you share the below entries of your ambari.ini and your ambari.properties?


[agent]
...
[security]
...
[heartbeat]
...
[logging]


Please revert