Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NoClassDefFoundError: org/apache/spark/internal/Logging

Labels:
avatar
author-rank Wuy
New Contributor

Created ‎07-30-2020 10:32 PM

i canYour post has been changed because invalid HTML was found in the message body. The invalid HTML has been removed. Please review the message and submit the message when you are satisfied.

3,666 Views
0 Kudos
2 REPLIES 2

avatar
author-rank Wuy
New Contributor

Created ‎07-30-2020 10:33 PM

I use spark-submit submit spark job to romotre ambari cluster with a client windows10  pc.
I get this problem .
my ambari cluster version is HDP-3.1.4.0 (spark 2.3.0)
my windows spark version 2.3.0,scala version 2.11.8
thanks a lot for help!!!!!!

3,664 Views
0 Kudos

avatar
author-rank Wuy
New Contributor

Created ‎07-30-2020 10:33 PM

 


PS D:\Spark\spark-2.3.0-bin-hadoop2.7\examples\jars> spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi spark-examples_2.11-2.3.0.jar 1000
2020-07-31 12:59:09 INFO SparkContext:54 - Running Spark version 2.3.0
2020-07-31 12:59:09 INFO SparkContext:54 - Submitted application: Spark Pi
2020-07-31 12:59:09 INFO SecurityManager:54 - Changing view acls to: dgbtds
2020-07-31 12:59:09 INFO SecurityManager:54 - Changing modify acls to: dgbtds
2020-07-31 12:59:09 INFO SecurityManager:54 - Changing view acls groups to:
2020-07-31 12:59:09 INFO SecurityManager:54 - Changing modify acls groups to:
2020-07-31 12:59:09 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(dgbtds); groups with view permissions: Set(); users with modify permissions: Set(dgbtds); groups with modify permissions: Set()
2020-07-31 12:59:09 INFO Utils:54 - Successfully started service 'sparkDriver' on port 62303.
2020-07-31 12:59:09 INFO SparkEnv:54 - Registering MapOutputTracker
2020-07-31 12:59:09 INFO SparkEnv:54 - Registering BlockManagerMaster
2020-07-31 12:59:09 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2020-07-31 12:59:09 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2020-07-31 12:59:09 INFO DiskBlockManager:54 - Created local directory at C:\Users\dgbtds\AppData\Local\Temp\blockmgr-50a0ec9b-c9c1-4a6a-b9a2-a880d621e169
2020-07-31 12:59:09 INFO MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2020-07-31 12:59:10 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2020-07-31 12:59:10 INFO log:192 - Logging initialized @2397ms
2020-07-31 12:59:10 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2020-07-31 12:59:10 INFO Server:414 - Started @2462ms
2020-07-31 12:59:10 INFO AbstractConnector:278 - Started ServerConnector@ec2bf82{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2020-07-31 12:59:10 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@69c43e48{/jobs,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@733037{/jobs/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7728643a{/jobs/job,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5167268{/jobs/job/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1cfd1875{/stages,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@28c0b664{/stages/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c444798{/stages/stage,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@436390f4{/stages/stage/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4d157787{/stages/pool,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@68ed96ca{/stages/pool/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6d1310f6{/storage,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3228d990{/storage/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54e7391d{/storage/rdd,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@50b8ae8d{/storage/rdd/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@255990cc{/environment,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51c929ae{/environment/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3c8bdd5b{/executors,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29d2d081{/executors/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@40e4ea87{/executors/threadDump,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58783f6c{/executors/threadDump/json,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a7b503d{/static,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@10b3df93{/,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@ea27e34{/api,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1dfd5f51{/jobs/job/kill,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3c321bdb{/stages/stage/kill,null,AVAILABLE,@Spark}
2020-07-31 12:59:10 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.124.2:4040
2020-07-31 12:59:10 INFO SparkContext:54 - Added JAR file:/D:/Spark/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar at spark://192.168.124.2:62303/jars/spark-examples_2.11-2.3.0.jar with timestamp 1596171550288
2020-07-31 12:59:10 WARN DomainSocketFactory:117 - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
2020-07-31 12:59:11 INFO TimelineClientImpl:297 - Timeline service address: http://slave1.spark:8188/ws/v1/timeline/
2020-07-31 12:59:11 INFO RMProxy:98 - Connecting to ResourceManager at master.spark/192.168.124.199:8050
2020-07-31 12:59:11 INFO Client:54 - Requesting a new application from cluster with 3 NodeManagers
2020-07-31 12:59:11 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (4864 MB per container)
2020-07-31 12:59:11 INFO Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead
2020-07-31 12:59:11 INFO Client:54 - Setting up container launch context for our AM
2020-07-31 12:59:11 INFO Client:54 - Setting up the launch environment for our AM container
2020-07-31 12:59:11 INFO Client:54 - Preparing resources for our AM container
2020-07-31 12:59:12 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://master.spark:8020/Spark_jars/spark-yarn/spark-yarn_2.11-2.3.0.jar
2020-07-31 12:59:12 INFO Client:54 - Uploading resource file:/C:/Users/dgbtds/AppData/Local/Temp/spark-10fa9f0a-eb88-4901-a1be-bcb6f53dbe76/__spark_conf__4369598958010522009.zip -> hdfs://master.spark:8020/user/dgbtds/.sparkStaging/application_1595604923172_0049/__spark_conf__.zip
2020-07-31 12:59:12 INFO SecurityManager:54 - Changing view acls to: dgbtds
2020-07-31 12:59:12 INFO SecurityManager:54 - Changing modify acls to: dgbtds
2020-07-31 12:59:12 INFO SecurityManager:54 - Changing view acls groups to:
2020-07-31 12:59:12 INFO SecurityManager:54 - Changing modify acls groups to:
2020-07-31 12:59:12 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(dgbtds); groups with view permissions: Set(); users with modify permissions: Set(dgbtds); groups with modify permissions: Set()
2020-07-31 12:59:12 INFO Client:54 - Submitting application application_1595604923172_0049 to ResourceManager
2020-07-31 12:59:13 INFO YarnClientImpl:273 - Submitted application application_1595604923172_0049
2020-07-31 12:59:13 INFO SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1595604923172_0049 and attemptId None
2020-07-31 12:59:14 INFO Client:54 - Application report for application_1595604923172_0049 (state: FAILED)
2020-07-31 12:59:14 INFO Client:54 -
client token: N/A
diagnostics: Application application_1595604923172_0049 failed 2 times due to AM Container for appattempt_1595604923172_0049_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2020-07-31 12:59:15.166]Exception from container-launch.
Container id: container_e07_1595604923172_0049_02_000001
Exit code: 1

[2020-07-31 12:59:15.168]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:854)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more


[2020-07-31 12:59:15.168]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:854)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more


For more detailed output, check the application tracking page: http://master.spark:8088/cluster/app/application_1595604923172_0049 Then click on links to logs of each attempt.
. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1596171553962
final status: FAILED
tracking URL: http://master.spark:8088/cluster/app/application_1595604923172_0049
user: dgbtds
2020-07-31 12:59:14 INFO Client:54 - Deleted staging directory hdfs://master.spark:8020/user/dgbtds/.sparkStaging/application_1595604923172_0049
2020-07-31 12:59:14 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2020-07-31 12:59:14 INFO AbstractConnector:318 - Stopped Spark@ec2bf82{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2020-07-31 12:59:14 INFO SparkUI:54 - Stopped Spark web UI at http://192.168.124.2:4040
2020-07-31 12:59:14 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:66 - Attempted to request executors before the AM has registered!
2020-07-31 12:59:14 INFO YarnClientSchedulerBackend:54 - Shutting down all executors
2020-07-31 12:59:14 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down
2020-07-31 12:59:14 INFO SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
2020-07-31 12:59:14 INFO YarnClientSchedulerBackend:54 - Stopped
2020-07-31 12:59:14 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2020-07-31 12:59:14 INFO MemoryStore:54 - MemoryStore cleared
2020-07-31 12:59:14 INFO BlockManager:54 - BlockManager stopped
2020-07-31 12:59:14 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2020-07-31 12:59:14 WARN MetricsSystem:66 - Stopping a MetricsSystem that is not running
2020-07-31 12:59:14 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2020-07-31 12:59:14 INFO SparkContext:54 - Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2020-07-31 12:59:14 INFO ShutdownHookManager:54 - Shutdown hook called
2020-07-31 12:59:14 INFO ShutdownHookManager:54 - Deleting directory C:\Users\dgbtds\AppData\Local\Temp\spark-0afb924a-da78-4d5f-b578-98df9ba1413c
2020-07-31 12:59:14 INFO ShutdownHookManager:54 - Deleting directory C:\Users\dgbtds\AppData\Local\Temp\spark-10fa9f0a-eb88-4901-a1be-bcb6f53dbe76
PS D:\Spark\spark-2.3.0-bin-hadoop2.7\examples\jars>

 

 

3,662 Views
0 Kudos
webinar banner
Announcements
Product Announcements
[ANNOUNCE] Cloudera on Public Cloud Runtime 7.2.18 is GA!
What's New @ Cloudera
Cloudera Operational Database (COD) supports enhancements to...
Community Announcements
March 2024 Community Highlights
Community Announcements
Celebrating 100,000 Members: A Journey of Growth and Connect...
Product Announcements
[ANNOUNCE] New Cloudera JDBC Connector for Apache Hive 2.6.2...
View More Announcements
meetups banner