If you are running spark applications on a YARN cluster then you do not need to directly allocate memory or machines to it. You can dedicate machines via labels, either for exclusive workloads or to handle heterogenous hardware better. If there is some application where latency and the ability to respond immediately to spikes in load matters, then dedicated labels work. For example; HBase in interactive applications. If different parts of the cluster have different hardware configurations (example: RAM, GPU, SSD for local storage), then labels helps you schedule jobs which need those features to only be executed on those machines Once you start using labels, the labelled hosts will be underutilized when that specific work isn't running: the permanent tradeoff. If you are just running queries on a cluster where that latency isn't so critical that you want to pre-allocate capacity on isolated machines, —then using queues makes is more efficient. You can also set up queue priorities and pre-emption, so your important spark queries can actually pre-empt (i.e. kill) ongoing work from lower-priority applications. What is important for Spark is having your jobs ask for the memory they really need: Spark likes a lot, and if the spark JVM/python code consumes more than was allocated to them in the Yarn container requests, the processes may get killed. ... View more

This looks like it's being triggered by the Spark -> timeline server integration, as ATS is going OOM when handling spark events. Which means its my code running in the spark jobs triggering this. What kind of jobs are you running? Short lived? Long-lived? Many executors? The best short-term fix is for you to disable the timeline server integration, and set the spark applications up to log to HDFS instead, with the history server reading it from there. The details of this are covered in Spark Monitoring 1. In the spark job configuration you need to disable the ATS publishing. Find the line spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService -delete it set the property spark.history.fs.logDirectory to an HDFS directory which must be writeable by everyone. For example, hdfs://shared/logfiles spark.eventLog.enabled true spark.eventLog.compress truespark.history.fs.logDirectoryhdfs://shared/logfiles 2. In the history server you need to switch to the filesystem log provider spark.history.providerorg.apache.spark.deploy.history.FsHistoryProviderspark.history.fs.logDirectoryhdfs://shared/logfiles The next spark release we'll have up for download (soon!) will log less events to the timeline server. Hopefully it will reduce the problems on the timeline server. There's also lots of work going on in the timeline server for future hadoop versions to handle larger amounts of data —by mixing stuff kept in HDFS with the leveldb data. for now, switching to the filesystem provider is your best bet ... View more

Note also you are going to get less IO bandwidth, as you move from 3 replicas (and hence 3 places to run code locally), to what is essentially a single replica, with the data spread across the network. Erasure coding is for best storing cold data where the improvements in storage density is tangible: it will hurt performance through -loss of locality (network layer) -loss of replicas (disk IO layer) -need to rebuild the raw data (CPU overhead) I don't think we have any figures yet on the impact. On a brighter note, 10GbE ToR switches are falling in price, so you could thing about going to 10 Gb on-rack, even if the backbone remains a bottleneck ... View more