Cloudera Community
Fawze
user rank
Master Collaborator

Re: Having Spark 1.6.0 and 2.1 in the same CDH

by Master Collaborator in Support Questions
‎07-26-2017 11:08 AM
‎07-26-2017 11:08 AM
[root@aopr-dhc001 ~]# alternatives --display spark2-conf spark2-conf - status is auto. link currently points to /liveperson/hadoop/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/etc/spark2/conf.dist /liveperson/hadoop/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/etc/spark2/conf.dist - priority 10 Current `best' version is /liveperson/hadoop/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/etc/spark2/conf.dist. ================= aopr-dhc001.lpdomain.com 10.16.144.131 9 Role(s) HDFS DataNode HDFS Gateway Impala Daemon Kudu Tablet Server Spark (Standalone) Gateway Spark (Standalone) Worker Spark Gateway YARN (MR2 Included) Gateway YARN (MR2 Included) NodeManager ... View more

Re: Having Spark 1.6.0 and 2.1 in the same CDH

by Master Collaborator in Support Questions
‎07-26-2017 06:49 AM
‎07-26-2017 06:49 AM
The spark gateway on all the nodes and the client configuration deployed. ... View more

Re: Having Spark 1.6.0 and 2.1 in the same CDH

by Master Collaborator in Support Questions
‎07-25-2017 11:40 AM
‎07-25-2017 11:40 AM
Yes ... View more

Re: Having Spark 1.6.0 and 2.1 in the same CDH

by Master Collaborator in Support Questions
‎07-24-2017 10:54 PM
‎07-24-2017 10:54 PM
Can you please help   [root@aopr-dhc001 bin]# spark2-shell Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:118) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:118) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:118) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:104) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more [root@aopr-dhc001 bin]# spark-shell Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 /_/ ... View more

Re: Having Spark 1.6.0 and 2.1 in the same CDH

by Master Collaborator in Support Questions
‎07-21-2017 01:39 PM
‎07-21-2017 01:39 PM
@srowen Thanks for your quick response.   In the cloudera manager i will have 2 services? and each has it's own different configurations?   For the developers is it seamless just releasing the job with the needed dependancies?   I like to have GA-1 regarding the version. ... View more

Having Spark 1.6.0 and 2.1 in the same CDH

by Master Collaborator in Support Questions
‎07-21-2017 01:16 PM
‎07-21-2017 01:16 PM
Hi Guys,   I'm planning to upgrade my CDH version to 5.10.2 and some of our Developers needs Spark 2.1 to use it in spark streaming.   I'm planning to manage the 2 versions using Cloudera manager, 1.6 will be intergrated one and the Spark 2.1 with parcels.   My questions:   1- Should i use the spark2 as a service? will they let me have 2 spark services, the reqular one and the spark 2.1 one?   2- is it preferable to istall the roles and gateways for spark on the same servers of the reqular one? i assume the history and spark server can be different servers and using different port for the history server, how it will looks like when i add 2 gateways on the same DN?   3- Is it compleicated to be managed?   4- Is there away that 2 versions conflicted and affecting the current Spark jobs? ... View more
Labels:

Re: How can I get some monitoring data from cloude...

by Master Collaborator in Support Questions
‎07-20-2017 09:04 PM
‎07-20-2017 09:04 PM
@mbigelow Do you collect the metrics using a code or a script?   i'm collecting alot of metrics from Yarn histroy server API and the impala API, intersting in the specific metrics you collected regarding the hive and impala usage analytics ... View more

spark error

by Master Collaborator in Support Questions
‎06-20-2017 07:47 AM
1 Kudo
‎06-20-2017 07:47 AM
1 Kudo
Hi,   Anyone familair with this Error in Spark job( this is from the AM Logs)   Container exited with a non-zero exit code 1 2017-06-20 10:37:02,785 [dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - Executor lost: 24 (epoch 3) 2017-06-20 10:37:02,784 [sparkDriver-akka.actor.default-dispatcher-43] INFO org.apache.spark.deploy.yarn.ApplicationMaster$AMEndpoint - Driver terminated or disconnected! Shutting down. svpr-dhc016.lpdomain.com:55642 2017-06-20 10:37:02,785 [sparkDriver-akka.actor.default-dispatcher-36] ERROR org.apache.spark.scheduler.cluster.YarnClusterScheduler - Lost executor 6 on svpr-dhc035.lpdomain.com: Executor heartbeat timed out after 145717 ms 2017-06-20 10:37:02,793 [sparkDriver-akka.actor.default-dispatcher-43] INFO org.apache.spark.deploy.yarn.ApplicationMaster$AMEndpoint - Driver terminated or disconnected! Shutting down. svpr-dhc016.lpdomain.com:34794 2017-06-20 10:37:02,795 [task-result-getter-0-SendThread(svpr-azk05.lpdomain.com:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server svpr-azk05.lpdomain.com/172.16.147.150:2181. Will not attempt to authenticate using SASL (unknown error) 2017-06-20 10:37:02,795 [Reporter] INFO org.apache.spark.deploy.yarn.YarnAllocator - Completed container container_e29_1497968230437_0016_01_000037 (state: COMPLETE, exit status: 1) 2017-06-20 10:37:02,795 [sparkDriver-akka.actor.default-dispatcher-36] INFO org.apache.spark.scheduler.TaskSetManager - Re-queueing tasks for 6 from TaskSet 18.0 2017-06-20 10:37:02,795 [sparkDriver-akka.actor.default-dispatcher-45] INFO org.apache.spark.storage.BlockManagerMasterEndpoint - Trying to remove executor 24 from BlockManagerMaster. 2017-06-20 10:37:02,807 [Reporter] INFO org.apache.spark.deploy.yarn.YarnAllocator - Container marked as failed: container_e29_1497968230437_0016_01_000037. Exit status: 1. Diagnostics: Exception from container-launch. Container id: container_e29_1497968230437_0016_01_000037 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:561) at org.apache.hadoop.util.Shell.run(Shell.java:478) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) ... View more
Labels:

Re: kudu service are getting down frequently

by Master Collaborator in Support Questions
‎06-15-2017 08:58 PM
‎06-15-2017 08:58 PM
We have NTP servers, wondering why only Kudu impacted. NTP and clock offset impacted a complete node and not a service. ... View more

Re: Running impala query with specified resosurce ...

by Master Collaborator in Support Questions
‎06-15-2017 03:22 PM
1 Kudo
‎06-15-2017 03:22 PM
1 Kudo
My bad: forgot that -q is used for a single query use.   I solved the issue by putting the 2 select statement at file and used the -f option.   Still investigate the issue of the pool. ... View more
Contact Me
Online
Offline
Last Visited
‎10-19-2023 10:11 PM
Community Statistics
Member Since ‎01-25-2017 01:09 PM
Last Visited ‎10-19-2023 10:11 PM
Posts 396
Kudos received 27