Member since
11-23-2015
4
Posts
0
Kudos Received
0
Solutions
11-13-2017
08:37 PM
hue version – hue-3.9.0 livy version – hue-livy-3.9.0 spark version – spark-1.6.1 [root@sn1 logs]# tailf hue-mapr-livy_server-sn1.out 11:46:59.596 [ForkJoinPool-1-worker-77] ERROR c.c.hue.livy.server.SessionServlet$ – internal error java.lang.NullPointerException: null at com.cloudera.hue.livy.spark.SparkSubmitProcessBuilder.com$cloudera$hue$livy$spark$SparkSubmitProcessBuilder$$fromPath(SparkSubmitProcessBuilder.scala:292) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.spark.SparkSubmitProcessBuilder.start(SparkSubmitProcessBuilder.scala:270) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.batch.BatchSessionYarn$.apply(BatchSessionYarn.scala:41) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.batch.BatchSessionYarnFactory.create(BatchSessionYarnFactory.scala:31) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.batch.BatchSessionFactory.create(BatchSessionFactory.scala:28) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.batch.BatchSessionFactory.create(BatchSessionFactory.scala:26) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.SessionManager.create(SessionManager.scala:52) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.SessionServlet$$anonfun$17$$anon$2$$anonfun$18.apply(SessionServlet.scala:113) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.SessionServlet$$anonfun$17$$anon$2$$anonfun$18.apply(SessionServlet.scala:112) ~[livy-assembly.jar:na] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[livy-assembly.jar:na] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1417) ~[livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262) ~[livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) ~[livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478) [livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [livy-assembly.jar:na] 10.10.70.180 – – – 09/Nov/2017:11:46:59 -0800 “POST /batches HTTP/1.1” 500 30
... View more
Labels:
11-13-2017
02:36 PM
@Roberto Sancho, I am getting the same exception. Please let me know how to resolve this issue When I try to run a sample python file using the spark notebook. It is running into below exception. Please let me know how can we resolve this issue. hue version – hue-3.9.0 livy version – hue-livy-3.9.0 spark version – spark-1.6.1 [root@sn1 logs]# tailf hue-mapr-livy_server-sn1.out 11:46:59.596 [ForkJoinPool-1-worker-77] ERROR c.c.hue.livy.server.SessionServlet$ – internal error java.lang.NullPointerException: null at com.cloudera.hue.livy.spark.SparkSubmitProcessBuilder.com$cloudera$hue$livy$spark$SparkSubmitProcessBuilder$$fromPath(SparkSubmitProcessBuilder.scala:292) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.spark.SparkSubmitProcessBuilder.start(SparkSubmitProcessBuilder.scala:270) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.batch.BatchSessionYarn$.apply(BatchSessionYarn.scala:41) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.batch.BatchSessionYarnFactory.create(BatchSessionYarnFactory.scala:31) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.batch.BatchSessionFactory.create(BatchSessionFactory.scala:28) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.batch.BatchSessionFactory.create(BatchSessionFactory.scala:26) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.SessionManager.create(SessionManager.scala:52) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.SessionServlet$$anonfun$17$$anon$2$$anonfun$18.apply(SessionServlet.scala:113) ~[livy-assembly.jar:na] at com.cloudera.hue.livy.server.SessionServlet$$anonfun$17$$anon$2$$anonfun$18.apply(SessionServlet.scala:112) ~[livy-assembly.jar:na] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[livy-assembly.jar:na] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1417) ~[livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262) ~[livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) ~[livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478) [livy-assembly.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [livy-assembly.jar:na] 10.10.70.180 – – – 09/Nov/2017:11:46:59 -0800 “POST /batches HTTP/1.1” 500 30
... View more
10-27-2017
09:37 AM
Hi All, Currently I am working on an ETL project, which uses spark largely for all of its computations.
For a POC I have tried integrating Spark with Hue and the integration worked fine.
But when I tried spark sql on the Hue notebook, the queries are executing fine, but it is executing in local mode always.
Another issue noticed is that the, the context is always running (Example : /apps/spark/local-1509061166070.inprogress).
The spark sql is connected to the spark thrift server to to access hive.
I am using Hue 3.12 and livy 3.12.
... View more
11-23-2015
03:04 PM
Hi i am getting the following exception when I tried to start the datanode service through cloudera manager.
I am using a centos machine. I have tried CDH5 installation.
I have created the directory /dfs/dn. the folder permissions looks as below.
drwxrwxrwx 2 root root 4096 Nov 23 15:53 dn
Invalid dfs.datanode.data.dir /dfs/dn :
EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:652)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:505)
at
Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/dfs/dn/"
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2424)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2397)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2289)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2336)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2513)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2537)
Request for your help.
... View more
