techsoln
Explorer
My Accepted Solutions
Show All

Re: spark2-submit using pyspark fails

by Explorer in Support Questions
‎05-01-2019 11:51 AM
‎05-01-2019 11:51 AM
This was failing since my python executable was not in .zip or .egg format. On creation of the executable in .zip format job was accepted. ... View more

Re: Is it impossible to run a Spark2 job with --ma...

by Explorer in Support Questions
‎05-01-2019 11:49 AM
‎05-01-2019 11:49 AM
looks like to run on client mode yoy need to set --master local in your spark2-submit command ... View more

Re: Is it impossible to run a Spark2 job with --ma...

by Explorer in Support Questions
‎04-30-2019 01:11 PM
‎04-30-2019 01:11 PM
From the URL https://www.cloudera.com/documentation/spark2/latest/topics/spark2_known_issues.html#ki_spark_submit_kerberos,   only blocking point is that you cannot use principal and keytab configuration with spark2.But definitely there could be more in it. I am also struggling to run pyspark2 using spark2-submit.     ... View more

spark2-submit using pyspark fails

by Explorer in Support Questions
‎04-29-2019 08:23 PM
‎04-29-2019 08:23 PM
Hi ,   I am upgrading from Spark 1.6.0 to Spark 2.1 on CDH 5.10 platform. I am trying to run spark2-submit command for python implementation and it is failing giving below error. Looks like it is expecting some path property while initilization and creating SparkContext object which is not happening. Error details are as below. Please suggest if any specific configuration is missing or required for spark2.   sc = SparkContext(conf=conf) File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__ File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 182, in _do_init File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 249, in _initialize_context File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1401, in __call__ File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127) at org.apache.hadoop.fs.Path.<init>(Path.java:135) at org.apache.hadoop.fs.Path.<init>(Path.java:94) at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:368) at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:481) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$13.apply(Client.scala:629) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$13.apply(Client.scala:627) at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:627) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:874) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:171) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:171) at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:236) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) ... View more
Labels:

Re: pyspark fails with python 3.6

by Explorer in Support Questions
‎04-29-2019 11:58 AM
‎04-29-2019 11:58 AM
Thanks, Agreed. I also found the bug details.   Based on the URL https://spark.apache.org/docs/1.6.0/#downloading you shared, it contains details which says it is compatible with 2.6+ and 3.1+ which is totally misleading since 3.6 is 3.1+   I have started working to upgrade my app to spark 2. Any suggestiosn on Spark 1.6 to Spark 2 migration guide on Cloudera cluster     ... View more

pyspark fails with python 3.6

by Explorer in Support Questions
‎04-29-2019 10:18 AM
‎04-29-2019 10:18 AM
Hi All,   We are currently using Spark 1.6 on CDH 5.10 platform. We are currently upgrading from python 2.7 to python 3.6 using anaconda distribution. While i try to do spark-submit in client mode the process is failing giving below error -   File "/apps/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/lib/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 381, in namedtuple TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'   We are not very clear about the cause of the failure. We have checked Spark documentation and it says that Spark 1.6.0 is compatible with python 3.0+.    Any thoughts or suggestions on this would be helpful ?   Thanks Hemil ... View more
Labels:

spark-submit fails giving FileNotFoundException

by Explorer in Support Questions
‎03-12-2019 07:40 AM
‎03-12-2019 07:40 AM
Hi,   I am running spark-submit job on yarn cluster during which it is uploading dependent jars in default HDFS staging directory  which is /user/<user id>/.sparkStaging/<yarn applicationId>/*.jar. On verification during spark-submit job, i see that jar is getting uploaded but spark-submit is failing with below error - file owner and group belongs to the same id using which spark-submit is performed. I also tried using configuration parameter spark.yarn.StagingDir but even that didn't helped.   Your professional inputs will help in addressing this issue.      Error stack trace - =========================   Diagnostics: File does not exist: hdfs://user/<user id>/.sparkStaging/<yarn application_id>/chill-java-0.5.0.jar java.io.FileNotFoundException: File does not exist: hdfs://user/<user id>/.sparkStaging/<yarn application_id>/chill-java-0.5.0.jar at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1257) at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1249) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1249) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)   Thanks Hemil ... View more
Labels:

Re: Zookeeper Setup for 2 Rack Data center

by Explorer in Support Questions
‎06-12-2017 01:42 PM
‎06-12-2017 01:42 PM
Thanks so much for your inputs. So is there any recommended solution (software based) to achieve rack failure fault tolerance capability with 2 racks. !! 🙂 ... View more

Re: Zookeeper Setup for 2 Rack Data Center

by Explorer in Support Questions
‎06-09-2017 02:13 PM
‎06-09-2017 02:13 PM
Hi, Thanks for your response. I understand zookeeper requirement about having the majority of known ZK nodes to be up. With the example you mentioned, it is fine when i have total 3 nodes and 1 node goes down, since 2F +1 = 3 and it is = our total nodes in quorum we had setup or you can also say that we still have more active nodes then failed nodes. But when we have inter rack communication(specially 2 rack), it is different. As i said, Let say - I have active 3 ZK nodes on Rack A and 4 on rack B. In this case my quorum size is 7 and if my Rack B goes down, i am left with only 3 ZK nodes active. In this case 2F+1 says that, i should have setup 9 nodes in my cluster (F = 4) whereas i have setup 7 nodes. So zookeeper cluster across 2 Racks is not going to satisfy above rule since no of failed nodes are more then active nodes. Is there anyways concept of Kafka mirroring or setting up isolated zookeeper cluster and then cross connect zookeeper cluster so that when rack B goes down, rack A zookeeper takes over the task help in this 2 rack communication ? ... View more

Re: Zookeeper Setup for 2 Rack Data center

by Explorer in Support Questions
‎06-09-2017 12:09 PM
‎06-09-2017 12:09 PM
I was referring to the concept of Kafka Mirroring - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330   If this is implemented then can it be used to achieve rack fault tolerance across 2 racks cluster setup ? My limit is to implement scalable & fault tolerant messaging layer with 2 Rack infrastructure. ... View more

Does Kafka Mirroring needs different zookeeper ens...

by Explorer in Support Questions
‎06-08-2017 09:35 PM
‎06-08-2017 09:35 PM
Hi, I understand that Kafka mirroring is replicating data across different data centers. I have some questions here as below - 1) Does these different data centers needs different zookeeper ensemble ? 2) Can these Kafka mirroring concept be a good option if we want to replicate data across 2 racks which are part of data center ? Thanks ... View more
Labels:

Re: Zookeeper Setup for 2 Rack Data center

by Explorer in Support Questions
‎06-08-2017 08:31 PM
‎06-08-2017 08:31 PM
Hi,   Thanks again. This gives me an idea of setting up Kafka cluster in VM environment which is then connected to my actual existing servers. How about Kafka mirroring ? Can these be useful concept of achieving fault tolerance in case of rack failure ? And how will zookeeper ensemble looks in it ? Will it be standalone on both the rack and the target rack zookeeper gets activated when source rack goes down ? ... View more

Re: Zookeeper Setup for 2 Rack Data center

by Explorer in Support Questions
‎06-08-2017 07:37 PM
‎06-08-2017 07:37 PM
Hi,   Thanks for your response. So can i say that Kafka which is managed zookeeper. is not a right messaging platform for 2 Rack network infrastructure to achieve seamless rack fault tolerant mechanism. Or any other mechanism exists ?   ... View more

Re: Can anyone explain Kafka rack awareness featur...

by Explorer in Support Questions
‎06-08-2017 06:12 PM
‎06-08-2017 06:12 PM
Hi, I am trying to understand the impact and design for zookeeper setup since Kafka is dependent on zookeeper for its operations. Zookeeper specifies 2F+1 no of nodes to be setup for reliable fault tolerance. Consider that If I have 2 racks and I setup 4 nodes on rack A and 5 on rack B (Total 9 zookeeper nodes) and rack B goes down (5 zookeeper nodes goes down). In that case with the requirement of 2F+1, it needs 11 zookeeper nodes where as I have only 9 nodes. So zookeeper in case of rack failure with higher no of nodes will not be able to sustain which will impact Kafka cluster behavior. Can you please provide your inputs on how to better setup zookeeper so that Kafka can work seamlessly in case of 2 rack infrastructure ... View more

Re: Can anyone explain Kafka rack awareness featur...

by Explorer in Support Questions
‎06-08-2017 05:58 PM
‎06-08-2017 05:58 PM
Hi, I am trying to understand the impact and design for zookeeper setup since Kafka is dependent on zookeeper for its operations. Zookeeper specifies 2F+1 no of nodes to be setup for reliable fault tolerance. Consider that If I have 2 racks and I setup 4 nodes on rack A and 5 on rack B (Total 9 zookeeper nodes) and rack B goes down (5 zookeeper nodes goes down). In that case with the requirement of 2F+1, it needs 11 zookeeper nodes where as I have only 9 nodes. So zookeeper in case of rack failure with higher no of nodes will not be able to sustain which will impact Kafka cluster behavior. Can you please provide your inputs on how to better setup zookeeper so that Kafka can work seamlessly in case of 2 rack infrastructure. ... View more

Zookeeper Setup for 2 Rack Data Center

by Explorer in Support Questions
‎06-08-2017 05:42 PM
‎06-08-2017 05:42 PM
Hi, We have a 2 rack data center and both the rack has multiple servers. We are looking to setup the zookeeper nodes on these servers and it should provide failover mechanism in case of any rack failure. Based on zookeeper admin requirement, it needs 2F+1 number of nodes and odd number of nodes in a quorum. So in my case let say, i setup 3 zookeeper nodes on rack A and 4 on rack B. So total nodes in quorum is 7. Based on 2F+1 if rack B goes down then F is 4 here and it needs 9 nodes. If Rack A goes down then it 7. So the scenario of complete rack failure(Rack B) with with larger number of nodes will not sustain. I want to understand if zookeeper is preferable solution to be used in case of applications with 2 rack datacenter. If yes, how can i build the infrastructure in such case. ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎05-12-2020 09:03 AM
Public Statistics
Date Registered ‎06-08-2017 08:33 AM
Last Visited ‎05-12-2020 09:03 AM
Total Messages Posted 19