Member since
09-29-2015
51
Posts
9
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2317 | 11-10-2018 12:51 AM | |
4366 | 12-14-2017 09:53 PM | |
1798 | 09-01-2017 09:10 PM | |
1836 | 05-01-2017 09:54 PM | |
814 | 02-08-2017 08:51 PM |
12-13-2018
09:46 PM
Can you check which node container_e04_1544715810515_0001_01_000005 got assigned to and get the NM logs from that host during this failure window?
... View more
11-10-2018
03:16 AM
That's because this keytab is used by YARN Service master which needs a service principal and not a user principal. Its all towards thwarting replay attacks.
... View more
11-10-2018
01:14 AM
Do you mean will we support principal of the format "user@EXAMPLE.COM"?
... View more
11-10-2018
12:51 AM
You can upload the keytab from any one host to hdfs and then set "keytab" value to that path, something like "hdfs:///user/user1/user1.keytab". Note, the principal_name in that case cannot contain _HOST anymore and has to be expanded to the hostname from where you chose to upload the keytab, so something like "user1/host1.example.com@EXAMPLE.COM".
... View more
05-31-2018
03:06 PM
Also, make sure you have updated all your nodes to HDP 2.6.5.
... View more
05-31-2018
03:05 PM
The "int getMemory()" API was marked deprecated in favor of the "long getMemorySize()" API. However seems like at the proto level the int API was removed and that's probably why it is throwing NoSuchMethodError. Update the code to use getMemorySize and try.
... View more
05-09-2018
03:34 PM
See if you can play with the RM cluster applications API with query params startedTimeBegin, startedTimeEnd, finishedTimeBegin and finishedTimeEnd to get what you need - e.g. http://<rm http address:port>/ws/v1/cluster/apps?startedTimeBegin=1525820200652&sta rtedTimeEnd=1525827204652&limit=100 Its important to put a limit in the query to avoid a runaway query potentially returning thousands of apps. Here is the documentation providing details - https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API
... View more
04-10-2018
08:25 PM
What kind of apps are you running? Can you check under the {application_id}/{container_id} directories under yarn.nodemanager.log-dirs if the apps are creating any logs? Are the apps creating a sub-directory under the container dir and then logging under that? Note, logs created under sub-directories are not aggregated.
... View more
04-07-2018
06:24 AM
Can you check the NM log in the host where at least one container of your job ran to see if you find any errors related to log-aggregation?
... View more
04-05-2018
11:13 PM
Can you change ownership of /tmp in HDFS to yarn:hadoop instead of hdfs:hadoop? Is it a secure cluster?
... View more
03-01-2018
07:14 PM
1 Kudo
You need to use a new tag, not one of the existing ones. Typically it is the "version" keyword followed by current timestamp. If you don't absolutely need to use the REST API or you don't want to deal with the version tag, you should use configs.sh/configs.py (the sh is not supported in some older Ambari versions). Sample get call is - /var/lib/ambari-server/resources/scripts/configs.py -a get -l <ambari_server_host> -n <cluster_name> -c capacity-scheduler -f /tmp/cs.json Sample output in /tmp/cs.json is - {
"properties": {
"yarn.scheduler.capacity.maximum-am-resource-percent": "0.4",
"yarn.scheduler.capacity.maximum-applications": "10000",
"yarn.scheduler.capacity.node-locality-delay": "40",
"yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator",
"yarn.scheduler.capacity.queue-mappings-override.enable": "false",
"yarn.scheduler.capacity.root.acl_administer_queue": "*",
"yarn.scheduler.capacity.root.capacity": "100",
"yarn.scheduler.capacity.root.queues": "Hive",
"yarn.scheduler.capacity.root.accessible-node-labels": "*",
"yarn.scheduler.capacity.root.Hive.acl_submit_applications": "*",
"yarn.scheduler.capacity.root.Hive.maximum-capacity": "100",
"yarn.scheduler.capacity.root.Hive.user-limit-factor": "4",
"yarn.scheduler.capacity.root.Hive.state": "RUNNING",
"yarn.scheduler.capacity.root.Hive.capacity": "100"
}
} Help is - /var/lib/ambari-server/resources/scripts/configs.py -h To perform the change you want to make, edit the file /tmp/cs.json with your desired change (update value of yarn.scheduler.capacity.root.Hive.user-limit-factor in your case) then use the "-a set" option with the same file. Sample cmd provided below - /var/lib/ambari-server/resources/scripts/configs.py -a set -l <ambari_server_host> -n <cluster_name>-c capacity-scheduler -f /tmp/cs1.json Note, you need to refresh queues to make this change take effect. You can do it by running rmadmin via command line - yarn rmadmin -refreshQueues Or, use the Ambari REST API - curl -u admin:admin -H 'Content-Type:application/json' -H 'X-Requested-By:ambari' -iX PUT -d '{"save": "true"}' http://<ambari-server>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/<view_instance_name>/resources/scheduler/configuration/saveAndRefresh You do not need to restart RM for capacity scheduler changes. However, if you make changes to other configs like yarn-site via configs.py you need to restart RM. You can do so by using Ambari REST APIs as shown below. Stop RM - curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"INSTALLED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER Start RM - curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"STARTED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER
... View more
02-28-2018
10:58 PM
@Veerendra Nath Jasthi, is your Ambari Server running on localhost?
... View more
02-21-2018
09:56 AM
These 2 pointers should help you get started - https://community.hortonworks.com/articles/3229/capacity-scheduler-users-can-submit-to-any-queue.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_yarn-resource-management/content/controlling_access_to_queues_with_acls.html
... View more
01-31-2018
05:22 PM
In the RM UI, you can click on the app id link for a spark job and follow the app-attempt link and then click on the logs link against the first container (typically the one ending with 0001). Check the AM logs there and see what you find.
... View more
12-14-2017
09:53 PM
When you use "-log_files spark.log", you are basically asking for files of exact same name (spark.log). Probably the files older than 2 days are rotated to different names and hence not showing up. Alternately, you can use "-log_files_pattern <Log File Pattern>" to fetch all files which match a pattern. So maybe something like this will fetch all old logs for you "-log_files_pattern spark*"
... View more
12-11-2017
06:15 PM
Once you go to YARN Configs tab you can search for those properties. In latest versions of Ambari these show up in the Settings tab (not Advanced tab) as sliders. You can increase the values by moving the slider to the right or even click the edit pen to manually enter a value.
... View more
12-08-2017
05:29 PM
You should be able to over-subscribe memory by setting yarn.nodemanager.resource.memory-mb to a value higher than the actual physical memory in your nodes. Alternately, you might want to check the value of yarn.scheduler.minimum-allocation-mb and lower it a bit to accommodate for more containers.
... View more
11-03-2017
04:45 PM
Spark Client is overriding the AM memory from 512mb to 896MB. Can you check Spark AM logs and see if the AM is overriding the container memory from 256MB to the higher value?
... View more
10-30-2017
04:56 PM
One container is always the AM (application master), that's why it is 3. Can you click on the application ID in the first row, and then click on the attempt ID link and then on each of the 3 container ID links to see how much memory each container is taking?
... View more
10-30-2017
04:36 PM
Make sure you restart all YARN services RMs, NMs after the change
... View more
10-30-2017
04:25 PM
Yes, set yarn.scheduler.minimum-allocation-mb to 512mb or less.
... View more
10-30-2017
03:59 PM
1 Kudo
The 3GB is the total memory across all containers. 4 apps in that screenshot show 3GB because they have 3 running containers. If you see the app in the 3rd row you will see 1 container only and hence 1024mb.
... View more
10-16-2017
04:07 PM
Not sure why this comment went all the way to the top. In case you have missed, I am pasting it here again - "Still not very sure if this is what you need, but this might help - In Ambari go to YARN configs -> Advanced yarn-site -> set yarn.resourcemanager.max-completed-applications to 50"
... View more
10-13-2017
06:01 PM
1 Kudo
Still not very sure if this is what you need, but this might help - In Ambari go to YARN configs -> Advanced yarn-site -> set yarn.resourcemanager.max-completed-applications to 50
... View more
10-12-2017
06:02 PM
You can click on the "Show entries" drop down on the top left and select 60 (sorry by default I think 50 does not exist). Then you can click on the "StartTime" column header to sort by start time. Clicking on the "ID" column also typically gives you the same order (which is the default anyway).
... View more
09-06-2017
06:21 PM
1- You can always click on the red-pen edit button and manually enter the exact value you want. 2- You can uncheck and ignore the changes recommended by Ambari or accept the changes. To view mapred, tez and hive config properties, make sure you are in the corresponding services tab.
... View more
09-05-2017
04:41 PM
yarn.scheduler.minimum-allocation-mb is exposed by Ambari in the "Settings" tab as a slider (shown below) and not in the Advanced tab. That's why you will not find it anywhere in the Advanced tab. The best way to find any config is to lookup in the search/filter bar on the top/right as shown below. When you search for "yarn.scheduler.minimum-allocation-mb" in that search box it will show you only the "Minimum Container Size (Memory)" slider as shown in the first image above.
... View more
09-01-2017
09:10 PM
- Updated the xml files For the above step, are you making the changes manually directly to the xml files or via Ambari UI in Configs tab? If you are making the changes outside Ambari, then Ambari always overwrites them when you start services via Ambari. Please make changes via Ambari Configs tab to be able to persist them across restarts. -Gour
... View more