Member since
03-14-2016
4721
Posts
1111
Kudos Received
874
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2024 | 04-27-2020 03:48 AM | |
4009 | 04-26-2020 06:18 PM | |
3231 | 04-26-2020 06:05 PM | |
2593 | 04-13-2020 08:53 PM | |
3846 | 03-31-2020 02:10 AM |
03-04-2020
03:21 PM
@san_t_o Can you please check few things: 1). Please verify what is the value set for "" property in the NodeManager option? (If it starts even for few seconds) # ps -ef | grep NodeManager Things to look for: 2). If above does not start due to the "" error then please check the permissions set for this directory: Example: # ls -ld /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
drwxrwxrwt. 8 hdfs hadoop 4096 Feb 25 07:23 /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
# ls -ld /var/lib/ambari-agent/tmp/
drwxrwxrwt. 12 ambari hadoop 4096 Mar 4 01:48 /var/lib/ambari-agent/tmp/ Why we wanted to check permissions on "/var/lib/ambari-agent/tmp/" and "/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir" directory because the "" is usually set to this directory so the yarn user should have proper read/write access on the directory listed here. Example: # grep 'JAVA_LIBRARY_PATH' /etc/hadoop/conf/yarn-env.sh
export JAVA_LIBRARY_PATH="${JAVA_LIBRARY_PATH}:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir" 3). Also please check if you see any "" related files here. Ideally those should be owned by "yarn" user like "yarn:hadoop" (hadoop is group). This directory and it's content should be writable be yarn user. Example: # ls -lart /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni*
-rwxr-xr-x. 1 yarn hadoop 752803 Dec 2 06:33 /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-2529926063314066012.8 Possible Cause: So if by mistake if you would have ever restarted YARN NodeManager /RM with "root" user then the permissions on those directories/files might have changed and it might not be writable by Yarn user. So please check the directory permissions if they are writable or not?
... View more
02-27-2020
07:40 PM
1 Kudo
@mike_bronson7 In addition to my previous comment: Ambari provides option to "Rolling Restart Kafka Brokers" (one by one). In the ambari UI when you click on Ambari UI --> Kafka --> Service Actions (drop down) --> "Restart Kafka Brokers" Then it basically shows the Rolling restart Settings like. You can decide how much time in your env one Kafka Broker full restart taks and then after how much time do you want other Kafka broker restart will be scheduled by Ambari. It can also be achieved using the API call as following: # curl -iskH "X-Requested-By: ambari" -u admin:admin -X POST -d '[{"RequestSchedule":{"batch":[{"requests":[{"order_id":1,"type":"POST","uri":"/clusters/NewCluster/requests","RequestBodyInfo":{"RequestInfo":{"context":"_PARSE_.ROLLING-RESTART.KAFKA_BROKER.1.3","command":"RESTART"},"Requests/resource_filters":[{"service_name":"KAFKA","component_name":"KAFKA_BROKER","hosts":"testnode2.example.com"}]}},{"order_id":2,"type":"POST","uri":"/clusters/NewCluster/requests","RequestBodyInfo":{"RequestInfo":{"context":"_PARSE_.ROLLING-RESTART.KAFKA_BROKER.2.3","command":"RESTART"},"Requests/resource_filters":[{"service_name":"KAFKA","component_name":"KAFKA_BROKER","hosts":"testnode3.example.com"}]}},{"order_id":3,"type":"POST","uri":"/clusters/NewCluster/requests","RequestBodyInfo":{"RequestInfo":{"context":"_PARSE_.ROLLING-RESTART.KAFKA_BROKER.3.3","command":"RESTART"},"Requests/resource_filters":[{"service_name":"KAFKA","component_name":"KAFKA_BROKER","hosts":"testnode5.example.com"}]}}]},{"batch_settings":{"batch_separation_in_seconds":"121","task_failure_tolerance":1}}]}}]' ttp://testnode1.example.com:8080/api/v1/clusters/NewCluster/request_schedules .
... View more
02-27-2020
07:34 PM
@mike_bronson7 Looks like you have asked a very similar query on the other thread : https://community.cloudera.com/t5/Support-Questions/amari-rest-API-how-to-stop-service-on-specific-host/m-p/290651
... View more
02-27-2020
04:34 PM
2 Kudos
@mike_bronson7 Error says : hostname=kafka01. Host not found So please check if the Hostname is correct (i means fully qualified hostnmae) Please compare it with the hostname listed in the following API call response. Just try to open this URL in the browser to see which hostname ambari is expecting: http://ambari_server_hostname:8080/api/v1/clusters/$CLUSTER_NAME/hosts/ .
... View more
02-27-2020
03:41 PM
3 Kudos
@mike_bronson7 Using API call Get the List of Hostnames where KAFKA_BROKERS are running: Example: # curl -iskH "X-Requested-By: ambari" -X GET -u admin:admin ttp://testnode1.example.com:8080/api/v1/clusters/NewCluster/services/KAFKA/components/KAFKA_BROKER?fields=host_components/HostRoles/hostname | grep host_name | awk -F ":" '{print $2}' | sed -e 's|["'\'']||g'
testnode2.example.com
testnode3.example.com
testnode5.example.com The API call to Start Kafka Broker on Node "testnode2.example.com" can be achieved as following: # curl -iskH "X-Requested-By: ambari" -u admin:admin -X PUT -d '{"RequestInfo":{"context":"Start Kafka Broker","operation_level":{"level":"HOST_COMPONENT","cluster_name":"NewCluster","host_name":"testnode2.example.com","service_name":"KAFKA"}},"Body":{"HostRoles":{"state":"STARTED"}}}' ttp://testnode1.example.com:8080/api/v1/clusters/NewCluster/hosts/testnode2.example.com/host_components/KAFKA_BROKER The API call to Stop Kafka Broker on Node "testnode2.example.com" can be achieved as following: # curl -iskH "X-Requested-By: ambari" -u admin:admin -X PUT -d '{"RequestInfo":{"context":"Stop Kafka Broker","operation_level":{"level":"HOST_COMPONENT","cluster_name":"NewCluster","host_name":"testnode2.example.com","service_name":"KAFKA"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}' ttp://testnode1.example.com:8080/api/v1/clusters/NewCluster/hosts/testnode2.example.com/host_components/KAFKA_BROKER . .
... View more
02-27-2020
03:46 AM
@cc1 There can be various reasons ...starting from - Low disk space, Low memory , - No jvm present , - Insufficient resources available on the namenode host, - Corrupted fsimage, - namenode port already in use - Corrupted Edits log file - Insufficient permissions to the NN startup user, - Incorrect configurations ... ...etc such various kind of reasons. .... Hence it will be good to first check the NameNode logs to find out what kind of error is it showing and then accordingly it can be troubleshooted.
... View more
02-24-2020
10:04 PM
1 Kudo
@mike_bronson7 Thank you for sharing the screenshot .. it is very clean now. Please replace the following: "$CLUSTER_NAME" with "'"$CLUSTER_NAME"'" "$service" with "'"$service"'" "_PARSE_.STOP.$service" with "'"_PARSE_.STOP.$service"'" In general, Replace any value which has $ABCD with a single quote and then quote mark as '"$ABCD"' So the over all change will be "$ABCD" ----> "'"$ABCD"'" Stop Kafka Service: # curl -iLv -u "admin:admin" -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"'"_PARSE_.STOP.$service"'","operation_level":{"level":"SERVICE","cluster_name":"'"$CLUSTER_NAME"'","service_name":"'"$service"'"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service Start Kafka Service # curl -iLv -u "admin:admin" -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"'"_PARSE_.START.$service"'","operation_level":{"level":"SERVICE","cluster_name":"'"$CLUSTER_NAME"'","service_name":"'"$service"'"}},"Body":{"ServiceInfo":{"state":"STARTED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service
... View more
02-24-2020
03:06 PM
@mike_bronson7 I think the only change you will need to make is the "KAFKA" service in Uppercase because "kafka" in lowercase will not exist as a service. Following is the example which i tested in my cluster and it works pretty fine. # export service=kafka (INCORRECT)
# export service=KAFKA
# curl -iLv -u "admin:admin" -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"_PARSE_.STOP.$service","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME","service_name":"$service"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service In order to start KAFKA service. # curl -iLv -u "admin:admin" -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"_PARSE_.START.$service","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME","service_name":"$service"}},"Body":{"ServiceInfo":{"state":"STARTED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service .
... View more
02-24-2020
11:53 AM
@Prabhu_Muppala As we see that the netstat command shows no oracle port 1521 opened. (no output means Oracle is not running on default listerer port 1521. [cloudera@quickstart ~]$ sudo netstat -tnlpa | grep 1521 . Also the following error indicates that your Oracle Database is not successfully running on "localhost:1521" Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470) So please verify why the oracle DB is not running and configured to use 1521 listener port? Please try to restart Oracle and then recheck if the port 1521 is opened? Or if you have configured to run Oracle on some other port? # netstat -tnlpa | grep $ORACLE_PROCESS_ID .
... View more
02-23-2020
05:13 PM
@pauljoshiva Can you please check what is the port for the "RegistryDNS Bind Port" ? Is it free of being used by some other process ? I am assuming that port is 53 (please change the port in the following command to verify if that port is free os being used)? # netstat -tnlpa | grep 53 If possible then can you try changing the port to something else and then see if that works? And check for "RegistryDNS Bind Port" Ambari UI --> Yarn --> Configs --> Advanced (tab) --> Registry Example: RegistryDNS Bind Port = 1553 Reference Threads: https://community.cloudera.com/t5/Support-Questions/YARN-Registry-DNS-Start-failed-Hortonworks-3/m-p/218794 https://community.cloudera.com/t5/Community-Articles/YARN-REGISTRY-DNS-Port-Conflict-Issue/ta-p/249117
... View more