Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

class not found error

class not found error

Master Collaborator

I have successfully created the package via SBT but when I try to run it I get the error :

"NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$"

I tried adding it to the CLASSPATH with no luck. I have put all the twitter*.jar files in the project "resources" folder so why SBT did not include them in the package? how can I fix this issue?

[root@hadoop1 scala-2.11]# spark-submit twitterpopulartags_2.11-1.0.jar
16/09/15 15:54:40 INFO SparkContext: Running Spark version 1.6.2
16/09/15 15:54:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/09/15 15:54:41 INFO SecurityManager: Changing view acls to: root
16/09/15 15:54:41 INFO SecurityManager: Changing modify acls to: root
16/09/15 15:54:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/09/15 15:54:41 INFO Utils: Successfully started service 'sparkDriver' on port 60782.
16/09/15 15:54:41 INFO Slf4jLogger: Slf4jLogger started
16/09/15 15:54:41 INFO Remoting: Starting remoting
16/09/15 15:54:42 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.100.44.17:41895]
16/09/15 15:54:42 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 41895.
16/09/15 15:54:42 INFO SparkEnv: Registering MapOutputTracker
16/09/15 15:54:42 INFO SparkEnv: Registering BlockManagerMaster
16/09/15 15:54:42 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-312f44a6-3ddd-437f-a95b-016a858d9612
16/09/15 15:54:42 INFO MemoryStore: MemoryStore started with capacity 511.5 MB
16/09/15 15:54:42 INFO SparkEnv: Registering OutputCommitCoordinator
16/09/15 15:54:42 INFO Server: jetty-8.y.z-SNAPSHOT
16/09/15 15:54:42 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/09/15 15:54:42 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/09/15 15:54:42 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.100.44.17:4040
16/09/15 15:54:42 INFO HttpFileServer: HTTP File server directory is /tmp/spark-0934b175-2e48-4d10-af21-b676ae0ec173/httpd-8eddc6c4-9f47-48a2-9361-d037b401a4e7
16/09/15 15:54:42 INFO HttpServer: Starting HTTP Server
16/09/15 15:54:42 INFO Server: jetty-8.y.z-SNAPSHOT
16/09/15 15:54:42 INFO AbstractConnector: Started SocketConnector@0.0.0.0:38556
16/09/15 15:54:42 INFO Utils: Successfully started service 'HTTP file server' on port 38556.
16/09/15 15:54:42 INFO SparkContext: Added JAR file:/root/TwitterPopularTags/target/scala-2.11/twitterpopulartags_2.11-1.0.jar at http://10.100.44.17:38556/jars/twitterpopulartags_2.11-1.0.jar with timestamp 1473969282380
16/09/15 15:54:42 INFO Executor: Starting executor ID driver on host localhost
16/09/15 15:54:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45046.
16/09/15 15:54:42 INFO NettyBlockTransferService: Server created on 45046
16/09/15 15:54:42 INFO BlockManagerMaster: Trying to register BlockManager
16/09/15 15:54:42 INFO BlockManagerMasterEndpoint: Registering block manager localhost:45046 with 511.5 MB RAM, BlockManagerId(driver, localhost, 45046)
16/09/15 15:54:42 INFO BlockManagerMaster: Registered BlockManager
16/09/15 15:54:43 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
16/09/15 15:54:43 INFO EventLoggingListener: Logging events to hdfs:///spark-history/local-1473969282412
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$
        at dot.state.fl.us.PrintTweets$.main(PrintTweets.scala:29)
        at dot.state.fl.us.PrintTweets.main(PrintTweets.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterUtils$
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 11 more
[root@hadoop1 scala-2.11]#
2 REPLIES 2
Highlighted

Re: class not found error

Contributor

@Sami,

I don't know for sure, this will solve your problem or not just take a look at it. Add the following in build.sbt file,

settings(
    name :="TweetStream",
    version :="1.0",
    scalaVersion :="2.11.7",
    mainClass in Compile:=Some("TweetStream"))

libraryDependencies ++=Seq("org.apache.spark"%%"spark-core"%"1.5.2","org.apache.spark"%%"spark-streaming"%"1.5.2","org.apache.spark"%"spark-streaming-twitter_2.11"%"1.5.2","com.google.code.gson"%"gson"%"2.7","org.twitter4j"%"twitter4j-core"%"3.0.3","org.twitter4j"%"twitter4j-stream"%"3.0.3")// META-INF discarding
mergeStrategy in assembly <<=(mergeStrategy in assembly){(old)=>{casePathList("META-INF", xs @ _*)=>MergeStrategy.discard
    case x =>MergeStrategy.first
   }}

And the assembly.sbt in the project subfolder

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

Re: class not found error

Master Collaborator

doesn't your solution also needs the github library and "git" utility ? I saw this solution on web and tried it but since I am behind the proxy git is not working.

If I can find an answer to the simple question , why I need to do sbt-assembly ? why the sbt does not include the jars placed in the resource folder and compile them together? or why the sbt does not download the twitter jars at the compile time , as its downloading dozens of other dependent jars I see during compile ?

Don't have an account?
Coming from Hortonworks? Activate your account here