@Kshitij Badani I work with Jayadeep, we have made sure of all the points that you mentioned. For point 5) we are able to use livy commands from command prompt. output: * upload completely sent off: 81 out of 81 bytes < HTTP/1.1 201 Created < Date: Thu, 29 Jun 2017 07:10:37 GMT < WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCwjfJg+Z8lYE1nmmiIPQB0gb3flO96lTm/elABws1vT02CKl+KcHkCHUObklGVgZwebtCN73AhZSQy60+d2LnYdWG < Set-Cookie: hadoop.auth="u=talend&p=talend@TRANSPORTATION-HDPDEV.GE.COM&t=kerberos&e=1498756237672&s=Kkj7P3Ig2g06wogRIzZQimhX1gQ="; HttpOnly < Content-Type: application/json; charset=UTF-8 < Location: /batches/34 < Content-Length: 100 < Server: Jetty(9.2.16.v20160414) < * Closing connection 0 {"id":34,"state":"starting","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":[]}[talend@ip-10-235-3-142 ~]$ curl -u: --negotiate -H "X-Requested-By: user" http://10.228.3.142:9889/batches/34 {"id":34,"state":"dead","appId":"application_1498720050151_0001","appInfo":{"driverLogUrl":"http://ip-10-235-0-154.ec2.internal:8188/applicationhistory/logs/ip-10-228-1-148.ec2.internal:45454/container_e82_1498720050151_0001_02_000001/container_e82_1498720050151_0001_02_000001/talend","sparkUiUrl":"http://ip-10-235-2-223.ec2.internal:8088/proxy/application_1498720050151_0001/"},"log":["\t ApplicationMaster RPC port: -1","\t queue: default","\t start time: 1498720241709","\t final status: UNDEFINED","\t tracking URL: http://10.228.3.142:9889/batches/34 user: talend","17/06/29 03:10:41 INFO ShutdownHookManager: Shutdown hook called","17/06/29 03:10:41 INFO ShutdownHookManager: Deleting directory /tmp/spark-c3c670df-280a-46e0-82fd-7ecc4efc5ef2","YARN Diagnostics:","User application exited with status 1"]}
The problem is only from livy zeppelin. The commands that we are trying in livy is as below: %livy.pyspark print ("Hello") The log that we are getting is as below: org.apache.zeppelin.livy.LivyException: Session 24 is finished, appId: null, log: [java.lang.Exception: No YARN application is found with tag livy-session-24-mbc0jh8y in 60 seconds. Please check your cluster status, it is may be very busy., com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$anonfun$1$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$anonfun$1$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$anon$1.run(Utils.scala:95)] at org.apache.zeppelin.livy.BaseLivyInterprereter.createSession(BaseLivyInterprereter.java:221) at org.apache.zeppelin.livy.BaseLivyInterprereter.initLivySession(BaseLivyInterprereter.java:110) at org.apache.zeppelin.livy.BaseLivyInterprereter.open(BaseLivyInterprereter.java:92) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Can you please help us to get this working.
... View more
