Member since
01-16-2014
336
Posts
43
Kudos Received
31
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2865 | 12-20-2017 08:26 PM | |
2905 | 03-09-2017 03:47 PM | |
2474 | 11-18-2016 09:00 AM | |
4101 | 05-18-2016 08:29 PM | |
3195 | 02-29-2016 01:14 AM |
05-25-2016
03:18 AM
Right. Just accepted your reply as a solution. thanks
... View more
05-25-2016
12:34 AM
Sidharth, Please create a new thread for a new issue, re-using an old thread could lead to strange comments when people make assumptions based on irrelevant information. For your issue: EPERM means that the OS is not allowing you to create the NM recovery DB and you have recovery turned on. Check the access to the recovery DB directory that you have configured. Wilfred
... View more
02-29-2016
01:14 AM
1 Kudo
I would use the spark action as much OOTB as possible leverage sharelib for since handles a number of things for you. You can use multiple versions of sharelib as described here check for overriding the sharelib. Wilfred
... View more
02-02-2016
03:30 AM
Hi Wilfred, Thanks for your response. I am using a same setup but with standard Spark 1.3. But I am seeing that even though I have set the minimum and maximum shares for a queue (both min and max memory set at 80% of available memory), if there is already a spark job running in a different queue taking 40% memory, it is never preempted! The job in the queue with 80% memeory asking for 70% of memory waits until job in the other queue is finished. It is weird that in the same setup the preemption works for Hadoop mapreduce jobs but not for spark jobs any idea? Thanks, Vinay
... View more
01-24-2016
03:09 PM
1 Kudo
A vcore is a virtual core. You can define it however you want. You could, as an example, define that a vcore is the processing power that is delivered by a 1GHz thread core. A 3GHz core would than be comparable to 3 vcores in the node manager. Your container request then needs to use multiple vcores which handles the difference in speed. Not a lot of clusters do this due to the administrative overhead and the fact that if the end users do not use the vcore correctly it can overload the faster machines. Wilfred
... View more
09-28-2015
08:48 PM
With the --files option you put the file in your working directory on the executor. You are trying to point to the file using an absolute path which is not what files option does for you. Can you use just the name "rule2.xml" and not a path. When you read the documentation for the files. See the important note at the bottom of the page running on yarn. Also do not use the Resources.getResource() but just use a open of a java construct like: new FileInputStream("rule2.xml") or something like it. Wilfred
... View more
09-28-2015
08:08 PM
To rule out that we have a custom jar issue can you run the pi example to make sure that the cluster is (not) setup correctly? We have documented how to run a spark application, with the example in our docs. The error that you show points to a classpath error and you can not find the Spark classes on your class path. WIlfred
... View more
09-22-2015
04:05 AM
To be completely in control I often recommend to use a shading tool for libraries like this. Using maven shade or gradle shadow to make sure that your code references your version is a shure fire way to get this working. When you build your project you "shade" the references in your code which means it always uses your version. Wilfred
... View more
09-21-2015
11:56 PM
To add onto Wilfred's response, what is your CDH version? HDFS does cache all positive entries for 5 minutes, but negative caching wasn't supported until CDH 5.2.0 onwards (via HADOOP-10755). See also http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-common/core-default.xml#hadoop.security.groups.negative-cache.secs (which lists negative caching's TTL default being 30s, vs. positive's 300s). NSCD does also do negative caching by default, which could explain why the problem is gone, depending on how many negative, WARN group-lookup failure entries you observe in the log.
... View more
09-02-2015
02:54 PM
We were finally able to resolve this issue. For us, we were using classpath precedence override's to use our own classpath lib jars before cloudera's common lib directory. This resulted in an older version of guice being loaded for certain actions which caused the errors in the application logs. We resolved this issue by resolving the dependency issue between our projects and cloudera's lib.
... View more