About Xuesong

Xuesong · ‎07-03-2014

Thanks maestro.

Xuesong · ‎07-03-2014

Thanks. How to calculate the similarity of the movies in Spark MLlib, like mahout's org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilityJob?

Xuesong · ‎06-23-2014

Thanks for Mr. Owen told me before that to using cloudera repo. My Spark ALS app are now successfully running on CDH4.6 . I change my pom.xml to cloudera-repos. <repositories> <repository> <id>cloudera.repo</id> <url>https://repository.cloudera.com/artifactory/cloudera-repos</url> <name>Cloudera Repositories</name> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> And change my dependencies in pom.xml <dependencies>      <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.2.0</version> </dependency>      <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-mllib_2.10</artifactId> <version>0.9.0-incubating</version> </dependency> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>2.10.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>0.9.0-cdh4.6.0</version> </dependency> </dependencies>

Xuesong · ‎06-20-2014

Hi Owen, how to run the jar of scala app. When I use "java -jar sparkalsapp-build.jar" , it look like this: [root@hadoop140 SparkALSApp]# java -jar sparkalsapp-build.jar Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Seq at org.hisense.intelligence.MovieLensALS.main(MovieLensALS.scala) Caused by: java.lang.ClassNotFoundException: scala.collection.Seq at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 1 more ************************************************************************************************************************************** When I use "scala -classpath sparkalsapp-build.jar org.hisense.intelligence.MovieLensALS", it look like this: [root@hadoop140 SparkALSApp]# scala -classpath sparkalsapp-build.jar org.hisense.intelligence.MovieLensALS Spark ALS for movie recommendation begin... java.lang.ClassNotFoundException: org.apache.spark.SparkConf at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.hisense.intelligence.MovieLensALS$.main(MovieLensALS.scala:37) at org.hisense.intelligence.MovieLensALS.main(MovieLensALS.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:71) at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31) at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:139) at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:71) at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:139) at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:28) at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:45) at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:35) at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:45) at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:74) at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:96) at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:105) at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala) **************************************************************************************************************** But my code is running good in spark-shell. Please tell me how to run the jar of scala app. Thanks.

Xuesong · ‎06-20-2014

Thanks.

Xuesong · ‎06-18-2014

Thanks. How to set set up Spark environment in Scala App for CDH4.6? How to Set Up parameters of sparkHome, master, and masterHostname in the following code? val jarFile = "/opt/SparkALSApp/out/artifacts/SparkALSApp_jar/SparkALSApp.jar" // "target/scala-2.10/movielens-als_2.10-0.0.jar" val sparkHome = “???” // "/root/spark" val master = “???” // Source.fromFile("/root/spark-ec2/cluster-url").mkString.trim val masterHostname = “???” // Source.fromFile("/root/spark-ec2/masters").mkString.trim val conf = new SparkConf() .setMaster(master) .setSparkHome(sparkHome) .setAppName("MovieLensALS") .set("spark.executor.memory", "8g") .setJars(Seq(jarFile)) val sc = new SparkContext(conf)

Xuesong · ‎04-28-2014

Thanks, Sean. You're a saint.

Xuesong · ‎04-28-2014

You are so kind. Thanks for your help.

Xuesong · ‎04-28-2014

Thanks. And I want to know how to reference the Cloudera repo and specify mahout distribution to 0.8-cdh5.0.0. When i changed the version to 0.8-cdh5.0.0, the build occurred error. Would you give a sample that using mahout version 0.8-cdh5.0.0? Now my pom.xml as: <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.xuesong.mymahout</groupId> <artifactId>myMahout</artifactId> <packaging>jar</packaging> <version>1.0-SNAPSHOT</version> <name>myMahout</name> <url>http://maven.apache.org</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <mahout.version>0.8</mahout.version> </properties> <dependencies> <dependency> <groupId>org.apache.mahout</groupId> <artifactId>mahout-core</artifactId> <version>${mahout.version}</version> </dependency> <dependency> <groupId>org.apache.mahout</groupId> <artifactId>mahout-integration</artifactId> <version>${mahout.version}</version> <exclusions> <exclusion> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty</artifactId> </exclusion> <exclusion> <groupId>org.apache.cassandra</groupId> <artifactId>cassandra-all</artifactId> </exclusion> <exclusion> <groupId>me.prettyprint</groupId> <artifactId>hector-core</artifactId> </exclusion> </exclusions> </dependency> </dependencies> </project>

Xuesong · ‎04-28-2014

Hi Sean, how to use oryx 0.5.1 to build recommendation app? Would you give me a sample?

Online	Offline
Last Visited	‎02-19-2017 08:37 PM

Member Since	‎01-26-2014 11:52 PM
Last Visited	‎02-19-2017 08:37 PM
Posts	35

Cloudera Community

Re: How to run the jar of scala app in Spark envir...

Re: How to calculate the similarity of movies in S...

How to calculate the similarity of movies in Spark...

Re: How to run the jar of scala app in Spark envir...

How to run the jar of scala app in Spark enviromen...

Re: How to set set up Spark environment in Scala A...

How to set set up Spark environment in Scala App f...

Re: How to use oryx to build recomendation app

Re: Using mahout 0.8 build recommendation app in c...

Re: Using mahout 0.8 build recommendation app in c...

How to use oryx to build recomendation app