About jsensharma

jsensharma · ‎02-03-2020

@asmarz Good to know that your original issue is resolved. However for any subsequent slightly different issue it is always better to open a new Community Thread that way the readers of this thread can easily find out One Error/Issue with one Solution. Multiple issues in a single thread can cause readers to get confused. . If your question is answered then, Please make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.

jsensharma · ‎02-03-2020

@mike_bronson7 If you just want to find out specific services like HDFS / HIVE /SPARK2 to know whether they require restart or not then you can do this using filtering option service_name.in(xxx,yyy,zzz) # curl -u admin:admin -H "X-Requested-By: ambari" -X GET "ttp://kerlatest1.example.com:8080/api/v1/clusters/KerLatest/host_components?HostRoles/stale_configs=true&HostRoles/service_name.in(HIVE,HDFS,SPARK2)" Various Options to get Stale Config Component/Service details. Similarly there are few other variants of the API call to find out and list the Services/Individual components which require restart. # curl -u admin:admin -H "X-Requested-By: ambari" -X GET "ttp://kerlatest1.example.com:8080/api/v1/clusters/KerLatest/host_components?HostRoles/stale_configs=true&fields=HostRoles/service_name,HostRoles/host_name" # curl -u admin:admin -H "X-Requested-By: ambari" -X GET "ttp://kerlatest1.example.com:8080/api/v1/clusters/KerLatest/host_components?HostRoles/stale_configs=true" # curl -u admin:admin -H "X-Requested-By: ambari" -X GET "ttp://kerlatest1.example.com:8080/api/v1/clusters/KerLatest/host_components?HostRoles/stale_configs=true&fields=HostRoles/service_name" How to restart All Stale Configurations using API call. # curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"RequestInfo":{"command":"RESTART","context":"Restart all required services","operation_level":"host_component"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true"}]}' "ttp://kerlatest1.example.com:8080/api/v1/clusters/KerLatest/requests" . Please change the Ambari Hostname/port and Cluster name in the above API calls. I have changed "http://" to "ttp://" Please correct that protocol prefix before using the above mentioned API calls.

jsensharma · ‎01-31-2020

@asmarz In order to clarify the port access, From Ambari host please check if the NameNode port and address is accessible? # nc -v $ACTIVE_NAMENODE_FQDN 8020 (OR) # telnet $ACTIVE_NAMENODE_FQDN 8020 The error which you posted usually indicates that before running the mentioned HDFS command you did not get a Valid kerberos ticket using "kinit" command. 20/01/31 16:31:19 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] . Most Possible Cause of above WARNING: If the port is accessible then please check if you are able to run the same hdfs command after getting a valid kerberos ticket. # klist -kte /etc/security/ambariservername.keytab # kinit -kt /etc/security/ambariservername.keytab ambariservername@REALM.COM # klist # export HADOOP_ROOT_LOGGER=DEBUG,console # hdfs dfs -ls / . And then try the same command using the "hdfs" headless keytab # kdestroy # klist -kte /etc/security/keytabs/hdfs.headless.keytab # kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-ker1latest@EXAMPLE.COM # klist # export HADOOP_ROOT_LOGGER=DEBUG,console # hdfs dfs -ls / *NOTE:* the "hdfs-ker1latest@EXAMPLE.COM" principal name may be different in your case so replace it with your own hdfs keytab principle Please share the output of the above commands. Also verify if all your cluster nodes has correct FQDN. . .

jsensharma · ‎01-31-2020

@Ioann Your API call does not look correct... the JSON data which you are posting is not right. Please try this kind of request. # curl -k -i -u admin:password -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop NodeManager","operation_level":{"level":"HOST_COMPONENT","cluster_name":"DEV","host_name":"slave01","service_name":"YARN"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}' https://ambari_server:8443/api/v1/clusters/DEV/hosts/slave01/host_components/NODEMANAGER . If it still fails in one of the env then please share the exact details about those envs: 1. Exact ambari server version from both the Envs: # ambari-server --version # ambari-server --hash 2. The Before making the API call on the failing cluster please make sure to put the ambari-server.log in "tail" mode so that we can see what kind of logging is appearing in amabri logs when you get following kind of INFO message Ignoring ServiceComponentHost as operation is not allowed, 3. Are you running this API call on both the envs as Ambari Admin user?

jsensharma · ‎01-30-2020

@mike_bronson7 Based on your details i do not think that your Postgres runs on 8080 port (that may be your Ambari Server port) So if you want to use "psql" utility to connect to Postgres Database then you should be using Postgres Port (default 5432). Example: # psql --host=`hostname -f` --port=5432 --username=ambari --dbname=ambari Password for user ambari: bigdata ambari=> select * from host_version; # psql --host=localhost --port=5432 --username=ambari --dbname=ambari Password for user ambari: bigdata ambari=> select * from host_version; . In order to know which port your Postgres is listening you can try to look at the followig places. # grep port /var/lib/pgsql/data/postgresql.conf # ps -ef | grep pgsql | grep ^postgres # netstat -tnlpa | grep 5432 # netstat -tnlpa | grep $POSTGRES_PID . Similarly you can confirm what is the Port in which your Ambari Server is listening for HTTP/HTTPS requests. # grep -e 'client.api.ssl.port\|port' /etc/ambari-server/conf/ambari.properties | grep api Example: # netstat -tnlpa | grep `cat /var/run/ambari-server/ambari-server.pid` tcp6 0 0 :::8440 :::* LISTEN 24715/java tcp6 0 0 :::8441 :::* LISTEN 24715/java tcp6 0 0 :::8080 :::* LISTEN 24715/java . .

jsensharma · ‎01-30-2020

@asmarz As we see the error like: Failed to run the Ambari Server org.apache.ambari.server.AmbariException: Ambari Server Kerberos credentials check failed. Check KDC availability and JAAS configuration in /etc/ambari-server/conf/krb5JAASLogin.conf . 1. So can you please let us know how did you enable Kerberos for Ambari Server ? or manually? 2. Do you have ambari-agent installed on the ambari server host? and Do you have the Kerberos clients installed on the ambari server host? # yum info krb5-libs # yum info krb5-workstation 3. Do you have the correct KDC/AD address defined inside the file : # ps -ef | grep AmbariServer | grep --color krb5.conf # cat /etc/krb5.conf . 4. Are you able to do "kinit" to get a valid kerberos ticket using the same detail mentioned in the file "/etc/ambari-server/conf/krb5JAASLogin.conf" # kinit -kt /etc/security/ambariservername.keytab ambariservername@REALM.COM # klist .

jsensharma · ‎01-30-2020

@mike_bronson7 How may connections do you see in the Postgres Stats? You can try running the following SQL query in your Postgres Db to see what is the output? https://www.postgresql.org/docs/9.2/monitoring-stats.html numbackends => Number of backends currently connected to this database. SELECT sum(numbackends) FROM pg_stat_database;

jsensharma · ‎01-30-2020

@mike_bronson7 - Something is really not right on the Host .where postgres is running ... may be some load. Have you checked if that host is running out of resources ? - Is this postgres Db shared by few other applications like Ranger/Hive/Oozie .etc - What kind of background jobs or other processes are running on this node? Here we see that sometimes the DB is taking more than 3 - 12 seconds delay to respond as well as the "Temporary failure in name resolution" as well as "PSQLException" issue is occurring intermittently. -- 11 Seconds Gap -- Thu Jan 30 14:48:35 UTC 2020 Checking Connection - 29 ERROR: Unable to connect to the DB. Please check DB connection properties. org.postgresql.util.PSQLException: The connection attempt failed. Thu Jan 30 14:48:46 UTC 2020 Checking Connection - 30 Connected to DB Successfully! -------- 6 Seconds Gap------- Thu Jan 30 14:50:10 UTC 2020 Checking Connection - 92 Connected to DB Successfully! Thu Jan 30 14:50:17 UTC 2020 Checking Connection - 93 Connected to DB Successfully! -------- 6 Seconds Gap------- Thu Jan 30 14:51:01 UTC 2020 Checking Connection - 130 Connected to DB Successfully! Thu Jan 30 14:51:07 UTC 2020 Checking Connection - 131 Connected to DB Successfully! -------- 12 Seconds Gap ------- Thu Jan 30 14:51:45 UTC 2020 Checking Connection - 158 hostname: Temporary failure in name resolution Connected to DB Successfully! Thu Jan 30 14:51:56 UTC 2020 Checking Connection - 159 Connected to DB Successfully! -------- 6 Seconds Gap------- Thu Jan 30 14:53:55 UTC 2020 Checking Connection - 254 Connected to DB Successfully! Thu Jan 30 14:54:01 UTC 2020 Checking Connection - 255 Connected to DB Successfully!

jsensharma · ‎01-30-2020

@mike_bronson7 I am still suspecting that your Postgres is not returning connection consistently... I mean it might be either responding very slow or sometimes it might not be responding connection..... (May be due to some LOAD on the system ..etc) So can you please try running the following kind of script to check the DB connection consistency in a for loop for some time ? Following is a smaple script "test_db_connection.sh" to test the connection consistently for 300 seconds # cat /tmp/test_db_connection.sh for i in {1..300} do echo `date` " Checking Connection - $i" /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver sleep 1 done . Please run the script as following and monitor if we are continuously getting the connection? # chmod 755 /tmp/test_db_connection.sh /tmp/test_db_connection.sh Thu Jan 30 14:41:10 UTC 2020 Checking Connection - 1 Connected to DB Successfully! Thu Jan 30 14:41:11 UTC 2020 Checking Connection - 2 Connected to DB Successfully! Thu Jan 30 14:41:13 UTC 2020 Checking Connection - 3 Connected to DB Successfully! . .

jsensharma · ‎01-30-2020

@mike_bronson7 What is the latest error in the "ambari-server.log" ? Also what is the output of the Test Connection command ?

Member Since	‎03-14-2016 01:07 PM
Last Visited
Posts	4,721
Kudos received	1096

Cloudera Community

Re: set Variable in ambari rest API

Re: how to stop Hive Metastore and HiveServer2 by...

Re: how to verify by ambari api the active/standby...

Re: Curl throws error when running allow snapshot

Re: ambari server + REASON: Server not yet listeni...

Re: Secure Webhdfs in Hadoop Hortonworks Cluster

Re: AMBARI REST API + how to verify if specific se...

Re: Secure Webhdfs in Hadoop Hortonworks Cluster

Re: Ambari rest api automation

Re: postgresql + ambari server failed to open port...

Re: Secure Webhdfs in Hadoop Hortonworks Cluster

Re: postgresql + ambari server failed to open port...

Re: postgresql + ambari server failed to open port...

Re: postgresql + ambari server failed to open port...

Re: postgresql + ambari server failed to open port...