Member since
03-14-2016
4721
Posts
1111
Kudos Received
874
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2720 | 04-27-2020 03:48 AM | |
| 5280 | 04-26-2020 06:18 PM | |
| 4445 | 04-26-2020 06:05 PM | |
| 3570 | 04-13-2020 08:53 PM | |
| 5377 | 03-31-2020 02:10 AM |
07-05-2017
04:26 PM
3 Kudos
@Daniel Müller This happens if you have changed the hostname of your cluster nodes after the ambari cluster installation. In thsi case because initially suppose the hostname was "hdp-3.novalocal" for the host so after starting the agent on that host it will be registered in the ambri DB with name "hdp-3.novalocal", But after few days if you will change the agent hostname to "hdp-3" then a new host will eb registered to the ambari cluster (even thoug the host is same but the hostname was different earlier)
The stop the Ambari Server # ambari-server stop
Please take ambari DB Dump. # pg_dump -U ambari ambari > /tml/ambari_bkp.sql .
We cleaned unwanted hosts from DB and delete those selected hosts. Get their "host_id" of those hosts which you want to clean.
Connect to ambari DB. # psql -U ambari ambari
Password: bigdata Queries: To find the host_id; select host_id from hosts where host_name='hdp-1.novalocal';
select host_id from hosts where host_name='hdp-2.novalocal';
select host_id from hosts where host_name='hdp-3.novalocal'; .
Using the above command we will get the "host_id". As we know that the above hosts need to be deleted, So delete them as following. Suppose the host_id is respectively 111,222,333 delete from execution_command where task_id in (select task_id from host_role_command where host_id in (111,222,333));
delete from host_version where host_id in (111,222,333);
delete from host_role_command where host_id in (111,222,333);
delete from serviceconfighosts where host_id in (111,222,333);
delete from hoststate where host_id in (111,222,333);
delete from hosts where host_name in ('hdp-1.novalocal');
delete from hosts where host_name in ('hdp-2.novalocal');
delete from hosts where host_name in ('hdp-3.novalocal');
delete from alert_current where history_id in ( select alert_id from alert_history where host_name in ('hdp-1.novalocal'));
delete from alert_current where history_id in ( select alert_id from alert_history where host_name in ('hdp-2.novalocal'));
delete from alert_current where history_id in ( select alert_id from alert_history where host_name in ('hdp-3.novalocal'));
The restart Ambari Server # ambari-server start . NOTE: Regarding the changing hostname issue I have written an article some time back, You should refer to the following article which explains why does it happen in Cloud environment and how to fix it. https://community.hortonworks.com/articles/42872/why-ambari-host-might-have-different-public-host-n.html .
... View more
07-05-2017
03:41 PM
@Gavin Duffy Ambari Server is a Java program , Where as Ambari Agent is a python program so if you have set the proxy property to the ambari server then it will not have any effect on agent behaviour. You should try the following: On all the ambari Agent Host, in the "~/.bash_profile" or "~/.profile" profile we can add the following: export http_proxy=http://YOURPOROXY:PORT .
... View more
07-05-2017
01:19 PM
@Gavin Duffy Can you please share the complete error from ambari-server.log? Are you able to do wget from the host where you are trying to start the registry service? With our Without Proxy? # wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.40.zip .
... View more
07-04-2017
06:25 AM
@Rohit Sharma Do you have a Kerberzed environment ? If yes then in order to see any DAG information please do the following in your Yarn config: yarn.admin.acl = yarn,dr.who,admin
yarn.acl.enable=false
Also check if your "tez-site.xml" file has the following property set properly ? tez.am.view-acls=* .
... View more
07-04-2017
04:15 AM
@Robin Dong
You can check the "/etc/ambari-server/conf/ambari.properties" file to know exactly which DataBase your ambari is using and then accordingly you can login to the DB host and run/check the requested commands or the DB admin can assist with the queries. Example: (In my Case it is Postgres)
# grep 'jdbc' /etc/ambari-server/conf/ambari.properties
custom.postgres.jdbc.name=postgresql-9.3-1101-jdbc4.jar
previous.custom.postgres.jdbc.name=postgresql-9.3-1101-jdbc4.jar
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat
.
... View more
07-04-2017
02:41 AM
@Robin Dong
You will need to have "mysql" client packages installed on the host where you are running this command: For example: # yum install mysql-community-client And then you can run the following command: # mysql -u ambari ambari -pbigdata .
... View more
07-04-2017
02:37 AM
@Robin Dong Looks like one of your host is causing the problem. Please check the following: 1). Check the "host_version" table and findout which host is not pointing to "2.5.3.0-37" SELECT * FROM host_version; 2). Login to the problematic host using SSH and then run the following command to find out if it really has the 2.5.3.0-37 packages installed or not? # hdp-select 3). If you see that the hdp-select command output shows that all the packages installed to that host is having the correct version then following the next step. 4). If you find the problematic host then find it's host_id from the "hosts" table and then check if it is pointing to OLD repo_version_id , if yes then update it to point to the current version. UPDATE host_version SET state='CURRENT' WHERE host_id='xx';
. Also you can update the "repo_version_id" for this host to point to the latest version of repo (if it is not set already) Please make sure that after this change the host is pointing to the repo_version_id mentioned in the "cluster_version" table which has state as "CURRENT".
. 5). After the DB changes (commits) ambari-server need to be restarted. # ambari-server restart
.
... View more
07-04-2017
01:45 AM
@Robin Dong Sometimes this Add Service UI can get hung when some of the Hosts of the cluster are in OUT_OF_SYNC state. Please check the ambari DB to findout those host states.
SELECT host_id, state FROM host_version WHERE state = "OUT_OF_SYNC";
SELECT host_id, host_name FROM hosts where host_id IN (SELECT host_id FROM host_version WHERE state = "OUT_OF_SYNC"); If you find any problematic host then please Enable the "Maintenance Mode" ON for those hosts first and then perform Add Service. .
... View more
07-04-2017
12:49 AM
@Robin Dong Did you notice any warning/error in ambari-server.log? (before or after the UI hung). The OS where the Ambari is running has enough memory (free -m). Also does ambari has enough Free space left. Following command will list the ambari memory details in run time. # $JAVA_HOME/bin/jmap -heap $AMBARI_PID Did you try opening the ambari UI in incognito mode browser? .
... View more