Member since
08-11-2014
481
Posts
92
Kudos Received
72
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2690 | 01-26-2018 04:02 AM | |
5662 | 12-22-2017 09:18 AM | |
2660 | 12-05-2017 06:13 AM | |
2961 | 10-16-2017 07:55 AM | |
8192 | 10-04-2017 08:08 PM |
06-07-2016
12:50 AM
Hi, thank you for quick reply. I am quite new in recommendation domain. What exactly mean latent features?
... View more
05-26-2016
12:58 PM
1 Kudo
Thanks @hubbarja. Spent the afternoon trying this out on the CDH 5.7.0 QuickStart VM, with a kerberos enabled cluster and Cloudera Kafka 2.0.0. I think perhaps I didn't quite phrase my question clearly, but what I was trying to ask was whether the spark-streaming-kafka client would support consuming from a Kafka cluster that has client SSL authentication required enabled. For anyone else who tries this, the summary is it won't work due to upstream Spark issue [SPARK-12177], which deals with support for the new Kafka 0.9 consumer / producer API. SSL, SASL_PLAINTEXT or SASL_SSL connections to Kafka all require use of the new API. In fact, this issue is referenced in the known issues released with CDH 5.7.0, I just didn't spot it in time. There's a pull request which appears to support SSL (but no form of Kerberos client authentication) in Github here, if anyone feels brave. Looking at the comments on the Spark ticket, it's going to be at least post Spark 2.0.0 release that this feature gets merged in, and probably not until 2.1.0. Back to the drawing board for me!
... View more
05-21-2016
10:42 AM
Yes you will certainly need to provide access keys for S3 access to work. I don't think (?) that would be a solution to a VerifyError, which is a much lower-level error indicating corrupted builds. Yes, it's expected that AWS SDK dependencies were updated along with the new Spark version in CDH 5.7. I think the current version should depend on jets3t 0.9, which is the one you want.
... View more
04-29-2016
06:32 AM
Thanks! Yes percent_rank() and window function together did the trick. A different way is to sort the column and pick the one that is in the middle. The results are close.
... View more
04-27-2016
06:22 PM
That's it. Thanks.
... View more
04-25-2016
05:45 AM
I have found the solution : var addedRDD : org.apache.spark.rdd.RDD[(String,Int)] = sc.emptyRDD
... View more
04-20-2016
12:37 PM
Thank you, now it makes a bit more sense.
... View more
01-13-2016
10:11 PM
but installing CDH 5.5 using tarball will have the spark and other components of hadoop too? I installed CDH 5.5 using tarball without cloudera manager. But can not see any jar of spark or any other component. Pls suggest how can I make use of inbuilt components of hadoop in CDH
... View more
11-21-2015
05:38 AM
Hi, I am trying to schedule a spark job using cron. I have made a shell script and it executes well on the terminal. However, when I execute the script using cron it gives me insufficient memory to start JVM thread error. Every time I start the script using terminal there is no issue. This issue comes when the script starts with cron. Kindly if you could suggest something.
... View more