Hi,
We are starting a java process from each executor of our streaming job. To start that java process we are using the following code:
SparkProcess process = new SparkProcess();
process.setExecDir(sparktemp);
process.setCommand("java");
process.addParameters("-Xmx" + xmx);
process.addParameters("-cp");
process.addParameters(sparktemp + "<jar_name>");
process.addParameters("<full class name>");
we are getting the following error:
Cannot run program "java" (in directory "<complete_path_of_jar>"): error=2, No such file or directory
at com.subex.roc.dataload.kafka2hive.StorageFunction$1.call(StorageFunction.java:631)
at com.subex.roc.dataload.kafka2hive.StorageFunction$1.call(StorageFunction.java:554)
at org.apache.spark.api.java.JavaRDDLike$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.api.java.JavaRDDLike$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.rdd.RDD$anonfun$foreachPartition$1$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.rdd.RDD$anonfun$foreachPartition$1$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:2069)
at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:2069)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot run program "java" (in directory "<complete_path_of_jar>"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at com.subex.sparkcommon.process.SparkProcess.execute(SparkProcess.java:50)
at com.subex.roc.dataload.kafka2hive.StorageFunction$1.call(StorageFunction.java:615)
... 13 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 15 more
We gave all permissions to the jar path and also to java path but still we are getting this error. The bashrc path has the java home and the path is also set to $JAVA_HOME/bin.
We are using HDP version 2.6.3
Please suggest what am I missing here. Also, please let me know should you need more info.
Thanks
Chandan
