Member since
08-04-2017
22
Posts
2
Kudos Received
0
Solutions
10-08-2017
04:57 PM
Hi, I am trying to submit a spark pi example in yarn-client mode and i am getting a weird error. HDP 2.4.2 and ambari 2.2.2 command : ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 4 --queue default lib/spark-examples*.jar 100 Error: INFO Client: Using the spark assembly jar on HDFS because you are using HDP, defaultSparkAssembly:hdfs://clusername/hdp/apps/2.4.2.0-258/spark/spark-hdp-assembly.jar
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$anonfun$setEnvFromInputString$1.apply(YarnSparkHadoopUtil.scala:440)
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$anonfun$setEnvFromInputString$1.apply(YarnSparkHadoopUtil.scala:438)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.setEnvFromInputString(YarnSparkHadoopUtil.scala:438)
at org.apache.spark.deploy.yarn.Client$anonfun$setupLaunchEnv$6.apply(Client.scala:643)
at org.apache.spark.deploy.yarn.Client$anonfun$setupLaunchEnv$6.apply(Client.scala:641)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:641)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:732)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:143)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1079)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1139)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) I am also getting same error when i try to restart the spark thrift server from ambari. Please help me fix this issue
... View more
Labels:
- Labels:
-
Apache Spark
08-16-2017
03:47 AM
Well I fixed this issue eventually by checking the ambari database hosts and one more table relating to hosts and have seen inconsistencies in the table.i have deleted the entries using the curl commands and restarted ambari and hosts tab works now. Before doing this I have checked the browser developer tools and seen the response where it is failing from the up part which also helped me to figure this issue. Hope it helps someone
... View more
08-15-2017
04:09 PM
yes i did that and when i click hive view it asks ldap password. My problem is my ldap password changes for every 4 hours, once i login in to hive view it does not ask for password anymore, well i guess hive view does not refresh when ldap password changes.how to fix this ?
... View more
08-15-2017
03:51 PM
we are currently using Hive view version 1.0 and ambari 2.2.2.0. I have integrated with LDAP and whenever i click on hive view it asks for credentials and i can successfully login and query it. But my ldap password changes for every 4 hours and hive view does not refresh or ask for my ldap credentials again which result in 401 error. Can some one please help me how should i fix this issue. It works fine as long as the initial password is valid. I have tried to clear the browser cache but did not work. How exactly does hive view authenticate
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
08-09-2017
12:18 AM
@nshelke How can we rectify these inconsistencies in ambari db
... View more
08-09-2017
12:16 AM
@nshelke I ran select * from hosts and i have some unwanted hosts in the table which defnitely show that it is inconsistent. Can you please help me as in what should i do?
... View more
08-09-2017
12:14 AM
@nshelke I ran the select * from hosts. I have found some hostnames which should not be present in the hosts table. How should i proceed in correcting them.
... View more
08-08-2017
11:51 PM
It did not help me, as my trnasport mode is http. my HS2 doesn't run on 10000.
... View more
08-08-2017
09:22 PM
There is no kerberos and ssl is not enabled for my case. Please help me
... View more