question Re: Does jar files missing for spark interpreter? in Support Questions

Does jar files missing for spark interpreter?

oriane_tchepgag — Wed, 22 Feb 2017 22:12:36 GMT

Hi,

I am facing a strange behaviour in sparK;

I am following the tutotial (lab 4 -spark analysis) but when executing the command the script is returned as you can see on the picture(red rectangle)

I have tables in My hive database default.

I checked the jar files in my spark interpreter and found jar files as you can see on picture sparkjarfiles.png (Are some jar files missing please?

Any suggestion?hivedisplay.png hicecontextdisplay.png

Re: Does jar files missing for spark interpreter?

jsensharma — Wed, 22 Feb 2017 22:32:20 GMT

@Oriane

By any chance do you have any extra line before the "%spark" or if there is any special character (my be while copying and pasting it might have come). Can you manually write those lines of %spark script freshly and then test again?

Re: Does jar files missing for spark interpreter?

bwalter1 — Wed, 22 Feb 2017 22:39:59 GMT

Side note: in HDP Zeppelin sqlContext defaults to hiveContext. So something like

%spark 
sqlContext.sql("show tables").collect.foreach(println)

should work.

Alternatively:

%sql
show tables

As Jay mentioned, the % needs to be in the first line

Re: Does jar files missing for spark interpreter?

jsensharma — Mon, 19 Aug 2019 10:33:14 GMT

Ideally it should work.

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Wed, 22 Feb 2017 22:47:54 GMT

Hi @Jay,

I checked but no extra line before "%spark"

I manually write but still facing the problemhicecontextdisplay-2.png

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Wed, 22 Feb 2017 22:59:53 GMT

Hi @Bernhard, I have tried both but facing the same problem.

Maybe jar files missing in my spark interpreter?

sqlnotwork.png sqlcontexttest.png

Re: Does jar files missing for spark interpreter?

bwalter1 — Wed, 22 Feb 2017 23:44:14 GMT

Does Zeppelin send to Spark Interpreter at all?

What is

%spark 
print(sc.version)

printing? No hiveContext necessary

Re: Does jar files missing for spark interpreter?

bwalter1 — Mon, 19 Aug 2019 10:33:06 GMT

and what are the Interpreter settings saying. Here are mine.

Note: a simple "python" in zeppelin.pyspark.python is also OK.

... by the way, I have the same libs in my zeppelin libs spark folder.

Have you tried to restart the interpreter?

Re: Does jar files missing for spark interpreter?

dkozlowski — Thu, 23 Feb 2017 14:59:39 GMT

@Oriane

Can you provide the following:

1. As @Bernhard Walter already asked, can you attach the screenshot of your spark interpreter config from Zeppelin UI

2. Create a new Notebook and run the below and send the output:

%sh
whoami

3. Can you attach the output of

$ ls -lrt /usr/hdp/current/zeppelin-server/local-repo

4. Is your cluster Kerberized?

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Thu, 23 Feb 2017 16:54:19 GMT

Hi @Bernhard, Excuse me not to have respected what you said before concerning the fisrt line.

the interpreter %jdbc(hive) and %sql are working well because the "show tables" display the result expected.

The problem i have is with %spark(respecting the first line) and I have errors as you can see on the joined picture.

In the interpreter binding, I have selected spark and save

sparkproblem.png

sparkproblem1.png

sparkconfiguration.png

I joined also the spark interpreter configuration

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Thu, 23 Feb 2017 17:19:46 GMT

localrepo.pngHi @Daniel

The %sh

whoami works well. It returns zeppelin. the problem I have is with %spark as said to @Bernhard.

I don't know what is happening this morning but not able to see the directoty hdp as you can see. Yesterday I could 😞

The proble is resolved, The ssh port was not correct so please see the result of the command on the pic "localrepo"

hdpdirectorydisappear.png

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Thu, 23 Feb 2017 17:33:39 GMT

Also facing the same problem with

sc.version or print(sc.version)

scversion.png

I have just install the sandbox on my laptop to do the "hello world" case.

I can I know if it is kerberized?

Re: Does jar files missing for spark interpreter?

dkozlowski — Thu, 23 Feb 2017 17:39:22 GMT

@Oriane

Do exactly this:

- in the new section type: %spark

- press <Enter> button

- type: sc.version

- press <Enter> button

Now, run it

Does this help?

I am asking as noticed that the copied code causing issues.

Re: Does jar files missing for spark interpreter?

jsensharma — Thu, 23 Feb 2017 17:48:23 GMT

@Oriane

I see that you are trying to do ssh on port 2122 which is not right ... many of basic commands will not work that way.

So please do ssh as following:

ssh -p 2222 root@127.0.0.1

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Thu, 23 Feb 2017 17:59:20 GMT

Thanks @jay. I have already noticed my error.

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Thu, 23 Feb 2017 18:08:22 GMT

Thanks @Daniel, I have already took @jay notice into account. I have no more paste the code since yesterday but type it by hand.

What i have done now is create a new notebook file

type %spark - press enter - type print(sc.version)- press enter , run it and I have a "prefix not found" display.

I have tried to open the log file from /var/log/zeppelin/zeppelin-interpreter-sh-zeppelin-sandbox.hortonworks.com.log (as you can see on the picture)scversion2.png scversion1.png logfile.png

I have also retstart spark interpreter again but still have the same "prefix not found" error.

Re: Does jar files missing for spark interpreter?

dkozlowski — Mon, 19 Aug 2019 10:32:58 GMT

@Oriane

For the "prefix not found" - double-check if you have spark interpreter binded in that notebook.

See my screenshot - Spark needs to be "blue"

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Thu, 23 Feb 2017 18:41:25 GMT

Oh! @Daniel ! you are wright with this "blue" detail!!!!

I though for the interpreter binding , spark should be "white"!

Please see this fantastic screen!

thanks a lot ! and keep helping newbies of newbies as me. !scversion3.png

Re: Does jar files missing for spark interpreter?

oriane_tchepgag — Thu, 23 Feb 2017 18:44:14 GMT

Thanks a lot @Jay! The problem resolved with a small notice of @Daniel Kozlowski concerning interpreter biding. The spark interpreter was "white" , I should have kept it "blue"

Thanks a lot !!

Re: Does jar files missing for spark interpreter?

dkozlowski — Thu, 23 Feb 2017 18:51:51 GMT

@Oriane

I am glad you have this working now.

If you believe I helped, please vote up my answer and select as best one 🙂