Created on 03-31-2018 06:11 AM - edited 09-16-2022 06:02 AM
Hi
I followed below steps when installing Spark:
1. downlaoded JDK 10
2. Set environment varibale "JAVA_HOME" under user variable. Update the PATH under user variable to %JAVA_HOME%\bin
3. downlaod spark (spark-2.3.0-bin-hadoop2.7). Copy all files into a folder called "spark".
4. downlaod winutils.exe from internet (GITHUB).
5. setup environemtn varibale "HADOOP_HOME" and "SPARK_HOME"
6. Add path %SPARK_HOME%\bin
When i run spark-shell i am getting below error message:
C:\spark>java -version java version "10" 2018-03-20 Java(TM) SE Runtime Environment 18.3 (build 10+46) Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode) C:\spark>spark-shell Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2464) at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:222) at org.apache.spark.deploy.SparkSubmit$.secMgr$lzycompute$1(SparkSubmit.scala:393) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:393) at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:401) at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:401) at scala.Option.map(Option.scala:146) at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:400) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:170) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3107) at java.base/java.lang.String.substring(String.java:1873) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52) ... 21 more C:\spark>
can anyone please help with above error.
Created 03-31-2018 03:58 PM
Update: The issue is resolved by using JDK 8.
Created 03-31-2018 03:58 PM
Update: The issue is resolved by using JDK 8.