There are two settings that you need to look at: -  yarn.scheduler.maximum-allocation-mb sets the maximum size of a container - yarn.nodemanager.resource.memory-mb sets the maximum size of the memory available on the node   When a request comes in for a container that is larger than the  maximum-allocation-mb it will be denied. The application can not be submitted.   If you have 240 GB in the host I would expect that the nodemanager would get about 200 GB of that if you just run Yarn on the node. That should allow you to run more than one large container as you have. However running with a 40GB container for MR seems a bit over the top: do you really need all that?   If you use DRF than you might not have a memory limitation but a vcores limitation. You have not mentioned anything about that side so I am not sure what you have configured and if that might be the problem or not. There are also things like the number of applications that can be run and the AM share which could be of influence on what you see.   There is a series of blog posts out that should help also with this it starts with: http://blog.cloudera.com/blog/2015/09/untangling-apache-hadoop-yarn-part-1/ Three parts are out currently part 4 is coming real soon...   If you need more help open a support case with us and we can work through setting up the scheduler with you.   Wilfred ... View more

Spark SQL is supported in CDH 5.5, with some limitations. One of those things that we do not support is the hive thriftserver, see: CDH 5.5 docs The thriftserver dependency on a version of hive and what we ship in CDH is still a problem.   Wilfred ... View more

You will need to change the build to pull in the right version as documented on the Spark pages. The maven repository information for CDH is documented in our generic docs.   You would probably get something like -Dhadoop.version=2.6.0-cdh5.4.0   Wilfred     ... View more

You should not ,and can not, rely on the joda version that the AWS SDK brings in. If they use a shaded version than you can not reach it and you would not see it. If they have an unshaded version then you need to shade your version. You need to declare it as your dependency and then shade your version in your build.   It is not the simplest thing to figure out, specially if you have never done it before, but after sorting it once it should not cost you anything and probably make upgrades of hadoop and maintenance of your application simpler.   Wilfred ... View more

If you need a version of a library that is already part of hadoop I would strongly recommend that you include your version of the library in a shaded form in your application. The shading will make sure that you get your version and that it will not interfere with existing versions.   We are currently writing a knowledge base article on how to do this and for know you will need to check online for "maven shade" or "gradle shadow" depending on how you build your application.   Wilfred ... View more

Hive on Spark is not officially supported and what you see is a one of those cases. Certain queries are slower, take more memory or fail. That is why it is not supported yet. We are working hard to fix and tune these use cases. Until that is done the only workaround is to fall back on the MR execution engine.   Wilfred ... View more

Yes it should use Spark when you do that. There is nothing else that you need to do to run Hive on Spark. Keep in mind that it is not officially supported for production.   Spark normally runs on top of YARN and you should thus see a Spark application in the RM that was run. You can also check the Spark JHS for the Spark data.   Wilfred ... View more

We highly recommend that you use the Spark action and not the shell action for Spark. Also make sure that you configure the gateways for Spark on the system.   If you need more detail you will need to provide a little more about what you are doing.   Wilfred ... View more

The thrift server in Spark is not tested, and might not be compatible, with the Hive version that is in CDH. Hive in CDH is 1.1 (patched) and Spark uses Hive 1.2.1. You might see API issues during compilation or run time failures due to that.   Wilfred ... View more

Node labels is not considered ready by Cloudera or even by the upstream community. The basis for node labels was added to Hadoop 2.6 with a large number of limitations. The only scheduler that currently implements node labels support is the CapacityScheduler. None of the other schedulers supports it yet. Cloudera recommends, for a number of reasons, that you use the FairScheduler in your cluster.   Setting up node labels is partially supported through the command line interface but it still requires manual steps and configuration. Support for labels is also limited to one (1) label per YARN application. Using labels requires you to add them on the command line when an application is submitted. MapReduce does not implement any of the node label support yet (MAPREDUCE-6304) in the current release.   Node labels due to its limited implementation can also cause a large increase in scheduling delays which makes using them counter productive. We are working with the community to make node labels ready for production but currently it is not there.   Wilfred ... View more

The NM loads its configuration  on startup and then reregisters with the RM. So the only thing that should be needed is update the configuration in CM for the node(s) and restart the NM service on the nodes. I have done that numerous times and it always works.   Wilfred ... View more

The codec is responsible for the reads and you will need to talk to the creator of the codec to provide you with the information on why this is happening.   Wilfred ... View more

We are aware of that upstream bug and found it during our internal performance testing, around the time we released CDH 5.5.1 It will be included in an upcoming CDH 5.5 release.   Wilfred ... View more

Use the yarn logs -applicationId APP_ID command to grab the executor logs so you can get some more detail on what is failing. APP_ID needs to be replaced with the application ID of your application in your example: application_1450777964379_0027   Wilfred ... View more

Those settings do not work on YARN so the no effect is expected. Check the Spark standalone doc. For YARN the cleanup should be automatic and triggered by a shutdown and proper clean up of the context.   Which versin of CDH are you running and how have you configured the shuffle?   Wilfred ... View more

The spark job server is not a Cloudera provided application. You will need to get support from the team that hosts the code at the main dev branch. That said I can see one huge problem: you try to use a job server version 0.4 which is for an older release of Spark (1.0.2) than you have in CDH 5.4 (1.3.x). Make sure that you use the proper version and fix your project compilation etc. Also Spark in CDH uses a base version of Spark and adds fixes on top of that. You might thus need a slightly different version of the job server than you think. It is all up to you to make sure it works for your use case and is stable. We are working on an equivalent job server as part of CDH.   Wilfred ... View more

No we do not support the thrift server as per the documentation: CDH 5.5 Spark release note Hive on Spark is also still in beta and we are finishing features as per Hive CDH 5.5 release note it is thus experimental and things might not work. We can not provide guidance on the road map for features that are not yet complete   Wilfred ... View more

With the --files option you put the file in your working directory on the executor. You are trying to point to the file using an absolute path which is not what files option does for you. Can you use just the name "rule2.xml" and not a path. When you read the documentation for the files. See the important note at the bottom of the page running on yarn.   Also do not use the Resources.getResource() but just use a open of a java construct like: new FileInputStream("rule2.xml") or something like it.   Wilfred ... View more

How to do this is documented here: running on yarn. You need to pass in a custom log4j.properties file. With rolling logs you will most likely lose the yarn log tracking and aggregation. I am not sure that this will properly work. The container will most likely keep pointing to the base file and never move to the rolled version or you will only be able to ever track the current one.   Wilfred ... View more

Files should be moved from done_intermediate to the done directory during the normal running of the JHS. Two things to check: - does the JHS show any errors in the logs? - run the following command on the host that runs the JHS: id -Gn mapred   it should show as an output: "mapred hadoop"   That is assuming that the JHS runs as the mapred user if it runs as another user replace the mapred in the id command.   Wilfred ... View more

Can you make sure the JHS is up and running and when you look for the job in the UI that it can find it? You can also try the command line to see if it is on HDFS?   yarn logs -applicationId APP_ID -appOwner USER_ID The APP_ID is the same as the job ID that you showed but with job replaced by application the USER_ID is the ID of user that ran the job   Wilfred ... View more