Member since
03-14-2016
4721
Posts
1111
Kudos Received
874
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2822 | 04-27-2020 03:48 AM | |
| 5472 | 04-26-2020 06:18 PM | |
| 4642 | 04-26-2020 06:05 PM | |
| 3692 | 04-13-2020 08:53 PM | |
| 5598 | 03-31-2020 02:10 AM |
02-27-2020
03:41 PM
3 Kudos
@mike_bronson7 Using API call Get the List of Hostnames where KAFKA_BROKERS are running: Example: # curl -iskH "X-Requested-By: ambari" -X GET -u admin:admin ttp://testnode1.example.com:8080/api/v1/clusters/NewCluster/services/KAFKA/components/KAFKA_BROKER?fields=host_components/HostRoles/hostname | grep host_name | awk -F ":" '{print $2}' | sed -e 's|["'\'']||g'
testnode2.example.com
testnode3.example.com
testnode5.example.com The API call to Start Kafka Broker on Node "testnode2.example.com" can be achieved as following: # curl -iskH "X-Requested-By: ambari" -u admin:admin -X PUT -d '{"RequestInfo":{"context":"Start Kafka Broker","operation_level":{"level":"HOST_COMPONENT","cluster_name":"NewCluster","host_name":"testnode2.example.com","service_name":"KAFKA"}},"Body":{"HostRoles":{"state":"STARTED"}}}' ttp://testnode1.example.com:8080/api/v1/clusters/NewCluster/hosts/testnode2.example.com/host_components/KAFKA_BROKER The API call to Stop Kafka Broker on Node "testnode2.example.com" can be achieved as following: # curl -iskH "X-Requested-By: ambari" -u admin:admin -X PUT -d '{"RequestInfo":{"context":"Stop Kafka Broker","operation_level":{"level":"HOST_COMPONENT","cluster_name":"NewCluster","host_name":"testnode2.example.com","service_name":"KAFKA"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}' ttp://testnode1.example.com:8080/api/v1/clusters/NewCluster/hosts/testnode2.example.com/host_components/KAFKA_BROKER . .
... View more
02-27-2020
03:46 AM
@cc1 There can be various reasons ...starting from - Low disk space, Low memory , - No jvm present , - Insufficient resources available on the namenode host, - Corrupted fsimage, - namenode port already in use - Corrupted Edits log file - Insufficient permissions to the NN startup user, - Incorrect configurations ... ...etc such various kind of reasons. .... Hence it will be good to first check the NameNode logs to find out what kind of error is it showing and then accordingly it can be troubleshooted.
... View more
02-24-2020
10:04 PM
1 Kudo
@mike_bronson7 Thank you for sharing the screenshot .. it is very clean now. Please replace the following: "$CLUSTER_NAME" with "'"$CLUSTER_NAME"'" "$service" with "'"$service"'" "_PARSE_.STOP.$service" with "'"_PARSE_.STOP.$service"'" In general, Replace any value which has $ABCD with a single quote and then quote mark as '"$ABCD"' So the over all change will be "$ABCD" ----> "'"$ABCD"'" Stop Kafka Service: # curl -iLv -u "admin:admin" -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"'"_PARSE_.STOP.$service"'","operation_level":{"level":"SERVICE","cluster_name":"'"$CLUSTER_NAME"'","service_name":"'"$service"'"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service Start Kafka Service # curl -iLv -u "admin:admin" -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"'"_PARSE_.START.$service"'","operation_level":{"level":"SERVICE","cluster_name":"'"$CLUSTER_NAME"'","service_name":"'"$service"'"}},"Body":{"ServiceInfo":{"state":"STARTED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service
... View more
02-24-2020
03:06 PM
@mike_bronson7 I think the only change you will need to make is the "KAFKA" service in Uppercase because "kafka" in lowercase will not exist as a service. Following is the example which i tested in my cluster and it works pretty fine. # export service=kafka (INCORRECT)
# export service=KAFKA
# curl -iLv -u "admin:admin" -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"_PARSE_.STOP.$service","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME","service_name":"$service"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service In order to start KAFKA service. # curl -iLv -u "admin:admin" -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"_PARSE_.START.$service","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME","service_name":"$service"}},"Body":{"ServiceInfo":{"state":"STARTED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service .
... View more
02-23-2020
05:13 PM
@pauljoshiva Can you please check what is the port for the "RegistryDNS Bind Port" ? Is it free of being used by some other process ? I am assuming that port is 53 (please change the port in the following command to verify if that port is free os being used)? # netstat -tnlpa | grep 53 If possible then can you try changing the port to something else and then see if that works? And check for "RegistryDNS Bind Port" Ambari UI --> Yarn --> Configs --> Advanced (tab) --> Registry Example: RegistryDNS Bind Port = 1553 Reference Threads: https://community.cloudera.com/t5/Support-Questions/YARN-Registry-DNS-Start-failed-Hortonworks-3/m-p/218794 https://community.cloudera.com/t5/Community-Articles/YARN-REGISTRY-DNS-Port-Conflict-Issue/ta-p/249117
... View more
02-20-2020
08:29 PM
2 Kudos
@mike_bronson7 As we see the error as: "Table Namespace Manager not fully initialized" 2020-02-21 03:33:49,284 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=15, retries=35, started=629725 ms ago, cancelled=false, msg=java.io.IOException: Table Namespace Manager not fully initialized, try again later
at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2693)
at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2915)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1686) Which indicates the AMS HBase master might have some issue. Can you please let us know when was "distributed Mode" AMS running fine last time? Or immediately after enabling AMS distributed mode it is not starting? Is it kerberos enabled environment? Can you please check the permission on the HDFS dir? (to verify if the ownership of this HDFS dir is setup correctly as "ams:hdfs") # su - hdfs -c 'hdfs dfs -ls /user/ams'
# su - hdfs -c 'hdfs dfs -ls /user/ams/hbase' If you still face any issue then may be you can try to change the Zookeeper ZNode for AMS and then try restarting AMS freshly. To change the "Zookeeper Znode Parent" property of AMS please try this. Ambari UI --> Amabri Metrics --> Configs --> "Advanced ams-hbase-site" --> "ZooKeeper Znode Parent" then change the value of the znode to something slightly different like "/ams-hbase-unsecure" to "/ams-hbase-unsecure1" ...etc and restart AMS and let us know if you see any error?
... View more
02-20-2020
07:28 PM
2 Kudos
@mike_bronson7 As the Metrics Service operation mode is already selected to "distributed" hence Ambari will make AMS aware that it needs to find that hbase.rootdir on HDFS. Following should be fine. hbase.rootdir=/user/ams/hbase
... View more
02-20-2020
02:47 PM
2 Kudos
@mike_bronson7 This does not look right? Ideally with HDFS HA name we do not use the Port number because "hdfsha" is not a hostname but just a logical name. hbase.rootdir=hdfs://hdfsha:8020/user/ams/hbase If you NameService name is "hdfsha" (defined in "Custom core-site" as "dfs.nameservices=hdfsha") then ideally you should be using the following in your AMS configuration in your "Advanced ams-hbase-site" hbase.rootdir=/user/ams/hbase . As your AMS mode is "distributed" hence AMS will automatically assume that the data is in HDFS and will be able to figure out the actual NameService name dynamically so we do not even need to specify "hdfs://hdfsha" there. After fixing the "hbase.rootdir" in AMS configs please kill and restart the AMS processes. Then specially check the AMS logs specially the following logs and please share if you notice any error ... please share the Full stacktrace. /var/log/ambari-metrics-collector/hbase-ams-master-*.log
/var/log/ambari-metrics-collector/hbase-ams-region-*.log
/var/log/ambari-metrics-collector/ambari-metrics-collector.log .
... View more
02-18-2020
10:19 PM
1 Kudo
@mark-gg As the Empty Base URL issue mentioned on the other thread which you referred as "AMBARI-25069" is already resolved in Ambari 2.7.4 and later so is it not a good idea to first upgrade Ambari to 2.7.4 (or Ambari 2.7.5 which is latest) and then try to register the desired version. Ambari Upgrade Guide: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.4.0/bk_ambari-upgrade-major/content/upgrade_ambari.html
... View more
02-17-2020
04:02 AM
@sharathkumar13 You can use the Cloudera Manager API call something like following to start/stop Kafka Service (or any desired service) In order to stop "kafka" service: # curl -iLv -u admin:admin -X POST --header 'Accept: application/json' 'ttp://dc-1.example.com:7180/api/v40/clusters/TestCluster/services/kafka/commands/stop' In order to start "kafka" Service # curl -iLv -u admin:admin -X POST --header 'Accept: application/json' 'ttp://dc-1.example.com:7180/api/v40/clusters/TestCluster/services/kafka/commands/start' . Please also take a look at the Cloudera Manager Swagger APIs. Cloudera Manager(CM) 6.0 introduces new Python API client cm_client based on Swagger. This new API client supports all CM API versions. https://cloudera.github.io/cm_api/docs/python-client-swagger/ . Here please replace the CM credentials and cluster name in CM host port in the above mentioned API calls.
... View more