Hello,   My Queue's Absolute used capacity is at 91.66667 % when i launch one application with spark-submit. What causes this ?   Thanks for clarifying this as i get an error if i submit 2 applications at the same time to my cluster getting the second app stuck in ACCEPTED with the message :    INFO Client: client token: N/A diagnostics: [Wed May 06 08:38:12 +0000 2020] Application is Activated, waiting for resources to be assigned for AM. Details : AM Partition = <DEFAULT_PARTITION> ; Partition Resource = <memory:98304, vCores:28> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 91.66667 % ; Queue's Absolute max capacity = 100.0 % ; Queue's capacity (absolute resource) = <memory:98304, vCores:28> ; Queue's used capacity (absolute resource) = <memory:90112, vCores:4> ; Queue's max capacity (absolute resource) = <memory:98304, vCores:28> ; ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: tab start time: 1588754291472 final status: UNDEFINED tracking URL: http://hadoop-master:8088/proxy/application_1588667960453_0011/ user: tab Which property of capacity-scheduler.xml affects this behaviour ? ... View more

Hello,   I have spark yarn client submitting jobs and when it does that, it creates a directory under my "HadoopTmp" which has files like for each submitted application: __spark_conf__8681611713144350374.zip __spark_libs__4985837356751625488.zip Is there a way these can be automatically cleaned? Whenever I submit a spark job I see new entries for these again in the same folder. This is flooding up my directory what should I set to make this clear automatically? Or should i manually delete them from HDFS ? ... View more

Hello !   I am performing extensive experiments over my 3-nodes (VMs) cluster. my VMs have a disk space of 50GB each, and checking the space available (on localhost:9870 (namenode's UI)) after 10 spark-submit application submissions reveal that the hard disks are almost plenty. How to delete that created data without restarting and reformatting the hdfs ?  I was thinking of a datanode clean up command to use here.   Thanks. ... View more

Hello,   I am trying to launch a spark app on Yarn over 2 nodes hadoop cluster. When i do it gets stuck with error message :    Waiting on ApplicationMaster container to launch and the app gets stuck in ACCEPTED status. When i check the ApplicationMaster Log in the resource manager UI, it says :      2021-10-25 12:22:53,788 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler 2021-10-25 12:22:53,960 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57482. 2021-10-25 12:22:53,960 INFO netty.NettyBlockTransferService: Server created on slaveVM1:57482 2021-10-25 12:22:53,960 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 2021-10-25 12:22:53,976 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, slaveVM1, 57482, None) 2021-10-25 12:22:53,976 INFO storage.BlockManagerMasterEndpoint: Registering block manager slaveVM1:57482 with 366.3 MiB RAM, BlockManagerId(driver, slaveVM1, 57482, None) 2021-10-25 12:22:53,976 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, slaveVM1, 57482, None) 2021-10-25 12:22:53,976 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, slaveVM1, 57482, None) 2021-10-25 12:22:54,194 INFO ui.ServerInfo: Adding filter to /metrics/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 2021-10-25 12:22:54,194 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7fe6122a{/metrics/json,null,AVAILABLE,@Spark} 2021-10-25 12:22:54,288 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030 2021-10-25 12:22:54,366 INFO yarn.YarnRMClient: Registering the ApplicationMaster 2021-10-25 12:22:56,433 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-10-25 12:22:58,467 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-10-25 12:23:00,502 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2021-10-25 14:32:25,915 INFO retry.RetryInvocationHandler: java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking ApplicationMasterProtocolPBClientImpl.registerApplicationMaster over null after 5 failover attempts. Trying to failover after sleeping for 27785ms.    in Yarn-site i set the resourcemanager.hostname  property in nodemanager to have a value of masterIP.   UPDATE : diagnostics: [lun ott 25 16:36:24 +0200 2021] Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:2096, vCores:1>; Queue Resource Limit for AM = <memory:0, vCores:0>; User AM Resource Limit of the queue = <memory:0, vCores:0>; Queue AM Resource Usage = <memory:0, vCores:0>; ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1635172583960 final status: UNDEFINED tracking URL: http://masterVM2:8088/proxy/applicati   Thanks for clarifying. ... View more

Hello,   I have configured & set Hadoop Cluster over 2 nodes and launch it along with Yarn like so :    On the master node : hdfs namenode -regular yarn resourcemanager   On the slave node :  hdfs datanode -regular yarn nodemanager And it shows through UI that there has been a connection established between the two machines that form the cluster. To note that start-dfs on the master node started both namenode and datanode even after setting slaves and hosts files. Now i submit an application (simple hello world) to Yarn : through this command : Spark-submit --class "main" --master yarn pathToJar   But i get the error  15/08/29 12:07:58 INFO Client: ApplicationManager is waiting for the ResourceManager  client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.hdfs start time: 1440864477580 final status: UNDEFINED  user: hdfs 15/08/29 12:07:59 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED) 15/08/29 12:08:00 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED) 15/08/29 12:08:01 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)...   What am i missing in my configuration ?     ... View more