- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
About the scheduler-capacity prop to launch many applications to the cluster
- Labels:
-
Apache Hadoop
-
Apache Spark
-
Apache YARN
Created on 01-23-2022 12:16 PM - edited 01-23-2022 02:07 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
My Queue's Absolute used capacity is at 91.66667 % when i launch one application with spark-submit. What causes this ?
Thanks for clarifying this as i get an error if i submit 2 applications at the same time to my cluster getting the second app stuck in ACCEPTED with the message :
INFO Client: client token: N/A diagnostics: [Wed May 06 08:38:12 +0000 2020] Application is Activated, waiting for resources to be assigned for AM. Details : AM Partition = <DEFAULT_PARTITION> ; Partition Resource = <memory:98304, vCores:28> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 91.66667 % ; Queue's Absolute max capacity = 100.0 % ; Queue's capacity (absolute resource) = <memory:98304, vCores:28> ; Queue's used capacity (absolute resource) = <memory:90112, vCores:4> ; Queue's max capacity (absolute resource) = <memory:98304, vCores:28> ; ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: tab start time: 1588754291472 final status: UNDEFINED tracking URL: http://hadoop-master:8088/proxy/application_1588667960453_0011/ user: tab
Which property of capacity-scheduler.xml affects this behaviour ?
Created 01-26-2022 09:54 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you check the userlimit of the queue and the max AM resource percentage ?
RM UI -> Scheduler -> Expand your queue( take screenshot and attach to this case)
