Thank you! ... View more

Thank you both for the quick replies! ... View more

Oops I meant to write S3 paths. Really, it's Hadoop and its APIs that supports / doesn't support S3. It should be built in to Hadoop distributions however. I believe you might need an s3a:// protocol instead of s3:// ... View more

Also, is there way to confirm csd file is properly deployed. Also, I don't see scala 11 libraries under /opt/cloudera/parcels/CDH/jars and only scala 10 libraries.    I heard that scala 10 and 11 both are installed with CDH 5.7 and later. Shouldn't scala 11 be available, Is this also cause for spark2 service not appearing.   I did all steps as mentioned and all steps did completely successfully, spark2 parcel is activated now.   Regards, Hitesh ... View more

Hi, thanks for the information! This is exactly what I expected but just wanted to make sure. ... View more

How do I query hive tables from spark 2.0 . Could you share the steps. ... View more

@MilesYao That may be. On that topic, I don't think anytime soon as Cloudera does not support many features that it does for Spark 1.6. I suspect that sometime post CDH 6 we will see Spark 2.x supplant Spark 1.x as the only version of Spark in CDH. Ah, I checked out HDP and see what you are getting at. It is really trivial on the difference. Cloudera asks you to put a file on the CM host and configure a separate parcel while HDP includes both Spark1 and Spark2 packages in the same repo. ... View more

Apologies, new thread created: https://community.cloudera.com/t5/Data-Science-and-Machine/Oryx2-ALS-recommender/m-p/56566 ... View more

The best way to deal with small files is to not have to deal with them at all. You might want to explore using Kudu or HBase as your storage engine instead of HDFS (Parquet). ... View more