The application gets assigned 57 or 62 containers to start within 20 seconds or so. At that point the log just stops. The AM should be doing the main amount of work in this case at the point you are at.   There is no scheduling problem on the RM side. If there would be a scheduling problem there you would not get any containers. Since you have containers assigned the AM should take over and start the containers on the NMs. There is no trace of that at all. This could be due to a number of things. Some points to check would be the connectivity between the AM and the NMs. One other thing to try is turn on debug logging on the AM and see if we show anything else on the AM side.   For now I would not suspect the RM at all, and not blame the scheduler in the RM.   Wilfred ... View more

Can you provide a bit more background for this issue? Which CDH version are you using? Do you have CM in the cluster or not? You will also need to explain far more about the Resource Manager configuration (scheduler and its config) and the node managers configs. Lots of resources is something that is open for many interpretations...   The last thing is your application requests: what kind of containers do you request for the different jobs you mentioned.   Thank you, Wilfred ... View more

Hi,   Could you please open a new thread next time for a new issue? It might at first look related but there is a good chance that it is something completely different and the old issue might distract from finding the solution quickly.   If you have some error mesages or more detail than please provide it so we can help you.   Wilfred ... View more

If you are not using Cloudera Manager you need to make sure that the configuration file is deployed to all nodes and in the proper place. CM handles this automatically for you (recent releases) but on older releases you might need to deploy the client config before starting the NM to make sure. The original message was posted 2 years ago so most likely that was in an old release of Cloudera Manager and the solution would have been to deploy client config and restart the NM.   NOTE: The config file must be located relative to the binary container-executor. If it is not found then you will get a message like that. Manual deployments are not simple for this file.   Wilfred ... View more

No this is not a known issue as far as I know. If you have a support contract please open a support case and we can look into it further for you.   Wilfred ... View more

Those directories should be cleaned up in current releases via either SPARK-7503 and or SPARK-7705. They are specific for yarn based setups. It should still happen automatically.   Wilfred ... View more

Umesh,   Those settings are for Spark standalone clusters only. I would strongly advise you not to run standalone but use Spark on Yarn. When you use Yarn the problem does not exist as Yarn handles it for you.   Wilfred ... View more

Since you are using the CDH distribution I hope you are using Cloudera Manager, that makes it much easier to manage and most things should have been configured for you.   You can access applications via the RM web UI there it should have links though to the container logs also. For Spark applications you should also have the Spark History Server up and running. You should be able to find the application there and links through to the container logs.   Logs are kept for days even for failed and finished applications. If you have log aggregation turned on you can use the yarn command to load the logs from HDFS. Otherwise the logs are kept on the local drives of the NodeManagers and you can check there.    Wilfred     ... View more

Look at the container log for the specific container " container_1476862885544_0131_01_000002" mentioned in the log. You are looking at NM logs, that will only tell you the general life cycle of the container. You need to look at what happens inside the container. Use the RM web UI and fins the application:  application_1476862885544_0131 and drill into the containers that are run for the application.   Wilfred ... View more

For your second issue. Depending on whether you have turned on log aggregation yarn.log-aggregation-enable or not you can increase the time to retain the logs to as long as you want by setting the appropriate retain setting: yarn.nodemanager.log.retain-seconds yarn.log-aggregation.retain-seconds For the first issue: if you really want to get the logs in multiple places then you should be able to do whatever log4j supports by changing the config. You can setup a container specific log configuration via the container-log4j.properties file. Passing that changed file in to the different applications is application dependent (see MAPREDUCE-6052 for example.   Wilfred ... View more

If you are starting with a cluster now I would strongly recommend that you use a CDH release much later than CDH 5.3. The later releases (CDH 5.8 or CDH 5.9) are far more stable than what you are trying to use now. Even if you stick with CDH 5.3 at least use the latest maintenance release.   Back to your question. There could be multiple things that cause your job to not start. First point to check would be the RM web UI and see what state the cluster and the scheduler is in. After that it depends on what the RM UI shows you...   Wilfred ... View more

There is a number of blog posts on our site that should help and answer all your questions. This one here should cover the questions on weights and things like it, but please read all of them.   Wilfred ... View more

That is a known issue: YARN-4022 The queue is removed just not from the display.   Wilfred ... View more

Your issue is here: running beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB. Give the container more space to run. You have run out of container space.   Since this is a Spark job and you are using pyspark the easiest solution would be to increase the overhead ( spark.yarn.executor.memoryOverhead)  that is used for this job in calculating the container size.   Wilfred ... View more

Hi,   You do not use either of these directly. The minimum settings are enforced by the scheduler: you can not request a container smaller than the minimum and the scheduler will round up anything smaller up to the minimum and give you the minimum container size. The increment is used for rounding up the container sizes. It makes the internal house keeping for the scheduler simpler. Neither setting influences the job settings directly, they are applied on top of the settings. You still need to set the resources that are needed for a job as part of the job configuration.   As examples: 1) request a container 600MB/1vcore: minimum size of a container is 1GB -> a 1GB/1vcore container is allocated 2) request a container 1200MB/1vcore: minimum size is 1GB, increment is 500MB -> a container of 1.5GB (rounded up to the next increment, the minimum is used as a base)   Same would happen for vcores.   For more info see Untangling Yarn blog series   Wilfred ... View more

No YARN-2246 is not in any release yet. I'll pull the patch into our release and we'll have it in an upcoming release.   Wilfred ... View more

For the application ownload the application logs and check what the error is in the logs:   yarn application -applicationId APPID -appOwner USERID Check the exit codes of the application and you should be able to tell in a bit more detail what is going on.   Wilfred ... View more

There is an open jira in Apache Oozie for that documentation issue: OOZIE-1283 So you can ignore that one and use the action extension as documented in the link given.   Wilfred   ... View more

You are referencing a real old jira which was resolved, and the action is still supported. In the oozie documentation we keep I am not able to find the deprecation message.   SSH actions has limitations in its usage, that is why we do not recommend it.   Wilfred ... View more

one container is one reducer or one mapper never more than one. There is no way to "limit" things inside a container since it is a one on one relationship.   Wilfred ... View more

It is here:   http://blog.cloudera.com/blog/2016/01/untangling-apache-hadoop-yarn-part-3/ ... View more