About Wilfred

chalimartines · ‎09-01-2015

Thank you, that was it. I missed this property.

Wilfred · ‎08-26-2015

This change will not happen. You can not change the scheduler without restarting the resource manager. It is not a job configurable setting but a server side setting only read on startup. Wilfred

sumit.nigam · ‎08-25-2015

I notice that memory on another CDH host is overcommitted. Of the total 62GB phy memory, the memory allotted to YARN containers is 40GB and that consumes the major chunk. I notice that my apps remain in ACCEPTED state despite requesting for only 1GB for each container. 1. Is this 40GB reserved by YARN at startup? 2. Plus, I assume that does not contribute to apps remaining in ACCEPTED state because it is from this pool of 40GB that my apps should get memory allocated, right? Thanks, Sumit

Grg · ‎08-20-2015

Hello, I still need to dig this but I will also check the MapFiles that are some Indexed SequenceFiles. I'll provide me feedback then 🙂 Greg.

Wilfred · ‎08-18-2015

Depending on how you have set up yarn hive should be part of the "allowed.system.users" list for the NM's that list will white list all system users below the "min.user.id". There is also a list of "banned.users" that lists all users that are not allowed to run containers. All these three need to be in sync to allow running a container. The hdfs user should not be allowed since it is the superuser and could circumvent the HDFS access permissions. When you execute a job from hue authentication is taken care of by hue. It will make sure that some kind of kerberos initialisation is performed. Wilfred

Wilfred · ‎08-18-2015

How many executors do you have when you run this? I see the same when I run it because it gets sent to each executor (2 in my case) Wilfred

SableFour · ‎08-12-2015

Thanks. This helped. By looking at the logs, I was able to fix the classpath issue on one of the nodes and giving enough executor memory helped the container not to get killed.

Wilfred · ‎07-27-2015

Cloudera Manager is not involved in YARN HA. You configure it through CM but after that the RMs handle the HA side using the zookeeper quorum. Logs will be the easiest and quickest to find an answer. Just a snipper around the failover should do it. Wilfred

Wilfred · ‎07-19-2015

In Spark a transformation works directly on the RDD. Transforms are implemented lazely and closely coupled to the RDDs. You can not use them separately. What you are looking for is a tool that can generate Saprk code for you based on the transformation rule. I don't think that something like that exists. Wilfred

Ayub · ‎07-02-2015

Thanks much Wilfred. In response to (1) reply - yes you will need to supply the setting every time. There is not something like: "if user == X the queue = Y", you can write your own rule for it if you wanted one. The rules can be added. -- I am just wondering how rules can be written to assign users to the queues in the rules, i don't any see users and queue specification section in rules. I mean to ask how is it possible to add rules to assign users to queues the want my requirement is i.e. if user == X the queue = Y"

Online	Offline
Last Visited	‎02-15-2023 08:41 PM

Member Since	‎01-16-2014 10:22 PM
Last Visited	‎02-15-2023 08:41 PM
Posts	336
Kudos received	43

Cloudera Community

Re: Shall we run multiple spark version jobs innoo...

Re: CompositeGroupsMapping

Re: Yarn Fair Scheduler Allocation file not found ...

Re: Odd behavior when pending mappers get stuck on...

Re: Have various Spark version running on the clus...

Re: Application master gets stuck when it gets mor...

Re: When i set the scheduler from program, it impa...

Re: Max apps per user.

Re: Build a HAR (Hadoop Archive) using Spark

Re: map reduce 2.0 throwing error after enabling k...

Re: How many times does the script used in spark p...

Re: spark-shell 1.3 errors out immediately with re...

Re: CDH 5.4.1 - Yarn Hight Availability does not w...

Re: Data transformations at Spark level

Re: YARN - Jobs not running under assigned queues