question Re: Zeppelin Notebook will not start in Sandbox 2.4 in Support Questions

Zeppelin Notebook will not start in Sandbox 2.4

Michael_Rife — Tue, 05 Apr 2016 04:05:36 GMT

Zeppelin Notebook worked for a couple queries in the Sandbox Tutorial Lab #6. It stopped working after a Step 6.3.1 (Hive Query). When I attempt to restart Zeppelin Notebook I get the following error in the /var/lib/ambari-agent/data/errors-495.txt file:

urllib2.URLError: <urlopen error [Errno 111] Connection refused>

Re: Zeppelin Notebook will not start in Sandbox 2.4

abajwa — Tue, 05 Apr 2016 04:18:56 GMT

Usually this happens when Zeppelin took longer to come up (e.g. if your VM is running out of resources). Can you give it a few min and see if status of Zeppelin turns green in Ambari

Re: Zeppelin Notebook will not start in Sandbox 2.4

sunile_manjee — Tue, 05 Apr 2016 10:49:25 GMT

Ali is right. What I do is shut down all unnecessary services on sandbox to run zeppelin

Re: Zeppelin Notebook will not start in Sandbox 2.4

Michael_Rife — Tue, 05 Apr 2016 19:52:47 GMT

I am new to Hadoop/Horton works. How do I know what are considered unnecessary services?

It was running before I did the Hive join query in the tutorial. Now it does not work. No new services starts or changed.

Re: Zeppelin Notebook will not start in Sandbox 2.4

Michael_Rife — Tue, 05 Apr 2016 19:52:49 GMT

I let it sit but nothing happened. I tried to restart the services. Same error. I shut down the VirtualBox machine and restart the Sandbox. Zeppelin-Notebook did not start. It fails at 100% and is remains red. I still get the following error:

urllib2.URLError: <urlopen error [Errno 111] Connection refused>

When I start the Sandbox it says that Zeppelin started with a green OK, but when I go into Ambari it does not start Zeppelin Notebook. Also, it now appears that Spark is not starting as well.

Re: Zeppelin Notebook will not start in Sandbox 2.4

sunile_manjee — Sun, 18 Aug 2019 12:21:53 GMT

Here is screen shot of services I have up and down. I use this configuration on my sandbox to run zep

Re: Zeppelin Notebook will not start in Sandbox 2.4

rmolina — Wed, 06 Apr 2016 01:11:32 GMT

Hi @Michael Rife typically seen this type of issues if not enough resources. Are you currently running another job? Did you lower any of the settings for the VM?

Re: Zeppelin Notebook will not start in Sandbox 2.4

Michael_Rife — Wed, 06 Apr 2016 02:10:09 GMT

Did not make any VM changes. A single table query worked. Then I tried the query joining two tables in the tutorial. No results were returned. After quite a while I refreshed the browser (F5) and at that point Zeppelin notebook stopped and I have not been able to get it started.

Re: Zeppelin Notebook will not start in Sandbox 2.4

Michael_Rife — Wed, 06 Apr 2016 02:27:35 GMT

Thanks for the suggestion. I stopped the processes you have stopped, but Zeppelin Notebook would not start with same error. I restarted the VM. Zeppelin shows green upon the OS startup, but it is not running when I look at the Amari dashbord.

Re: Zeppelin Notebook will not start in Sandbox 2.4

Michael_Rife — Sun, 18 Aug 2019 12:21:45 GMT

I compared the start up logs from when Zeppelin last successfully started and the last time I attempted to start it and it did not start. The left side is the successful startup log. The right is the unsuccessful startup log.

Re: Zeppelin Notebook will not start in Sandbox 2.4

sunile_manjee — Thu, 07 Apr 2016 09:30:51 GMT

@Michael Rife

On ambari home page what does it show in terms of resource usage on your sandbox? Are their any orphan jobs running?

Re: Zeppelin Notebook will not start in Sandbox 2.4

cdantonio — Thu, 28 Apr 2016 02:51:57 GMT

@Michael Rife, I don't know if you solved this, but I just had the issue and solved it for myself.

Reading the source code Zeppelin appears to be trying to connect to itself at that point, and since it hasn't started correctly it isn't able to start. I connected to the VM and confirm that nothing is listening on the Zeppelin port, so it's reasonable that it would fail to connect.

I look at the log for Zeppeloin and noticed multiple stack traces for an exception

javax.servlet.ServletException: Resource class org.apache.zeppelin.server.ZeppelinServer can not be instantiated due to InvocationTargetException

and went off Googling. It appears that this can be caused by an empty notebook file. So I looked for one and found it

[root@sandbox notebook]# find . -ls | more
2367730    4 drwxr-xr-x  17 zeppelin hadoop       4096 Apr 27 17:29 .
2371541    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2ANTDG878
2371542   16 -rw-r--r--   1 zeppelin hadoop      13609 Feb 26 20:47 ./2ANTDG878/note.json
2371561    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2BEQE47HR
2371562    4 -rw-r--r--   1 zeppelin hadoop       3546 Feb 26 20:47 ./2BEQE47HR/note.json
2371564    4 -rwxr-xr-x   1 zeppelin hadoop        326 Feb 26 20:47 ./update_all_notebooks.sh
2371557    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2BB5CUPUW
2371558 1048 -rw-r--r--   1 zeppelin hadoop    1071678 Feb 26 20:47 ./2BB5CUPUW/note.json
2371555    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2BAVUZ7NA
2371556  100 -rw-r--r--   1 zeppelin hadoop     102307 Feb 26 20:47 ./2BAVUZ7NA/note.json
2371539    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2ANT56EHN
2371540   16 -rw-r--r--   1 zeppelin hadoop      14906 Feb 26 20:47 ./2ANT56EHN/note.json
2371545    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2AS5TY6AQ
2371546   52 -rw-r--r--   1 zeppelin hadoop      52672 Feb 26 20:47 ./2AS5TY6AQ/note.json
2371537    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2A94M5J1Z
2371538   16 -rw-r--r--   1 zeppelin hadoop      13057 Feb 26 20:47 ./2A94M5J1Z/note.json
2371543    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2APFTN3NY
2371544 1292 -rw-r--r--   1 zeppelin hadoop    1319772 Feb 26 20:47 ./2APFTN3NY/note.json
2359541    4 drwxr-xr-x   2 zeppelin hadoop       4096 Apr 27 18:12 ./2BM4EHNF7
2359543    0 -rw-r--r--   1 zeppelin hadoop          0 Apr 27 18:12 ./2BM4EHNF7/note.json
2359427    4 drwxr-xr-x   2 zeppelin hadoop       4096 Apr 27 17:20 ./2BKDJ7EJQ
2359540   16 -rw-r--r--   1 zeppelin hadoop      16137 Apr 27 17:20 ./2BKDJ7EJQ/note.json
2371535  540 -rw-r--r--   1 zeppelin hadoop     549775 Mar 14 15:07 ./notebooks.zip
2371549    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2B48PF7SN
2371550   96 -rw-r--r--   1 zeppelin hadoop      96143 Feb 26 20:47 ./2B48PF7SN/note.json
2371559    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2BBBW75VS
2371560 1188 -rw-r--r--   1 zeppelin hadoop    1213695 Feb 26 20:47 ./2BBBW75VS/note.json
2371553    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2B522V3X8
2371554  268 -rw-r--r--   1 zeppelin hadoop     274336 Feb 26 20:47 ./2B522V3X8/note.json
2371563    8 -rw-r--r--   1 zeppelin hadoop       6433 Feb 26 20:47 ./README.md
2371547    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2B21B3AYC
2371548   76 -rw-r--r--   1 zeppelin hadoop      77501 Feb 26 20:47 ./2B21B3AYC/note.json
2371551    4 drwxr-xr-x   2 zeppelin hadoop       4096 Feb 26 20:47 ./2B4TWGC8M
2371552   48 -rw-r--r--   1 zeppelin hadoop      46910 Feb 26 20:47 ./2B4TWGC8M/note.json

I moved the directory with the offending file (./2BM4EHNF7) off to /tmp and it worked.

A quick way to do this is to SSH into the sandbox and use a one liner to move the files out:

$  ssh root@127.0.0.1 -p 2222
root@127.0.0.1's password: 
Last login: Wed Apr 27 18:52:40 2016 from 10.0.2.2
[root@sandbox ~]# BAD_FILE=`find /usr/hdp/current/zeppelin-server/lib/notebook -name "*.json" -size 0` ; mv `dirname $BAD_FILE` /tmp/

Re: Zeppelin Notebook will not start in Sandbox 2.4

Michael_Rife — Thu, 28 Apr 2016 22:57:15 GMT

@Chuck D'Antonio, I had the same problem. I had an empty notebook (empty note.json file). I moved the directory to /tmp and the Zeppelin Notebook server started. I assume that I can now drop the directory I moved to /tmp.

Thanks!!

Re: Zeppelin Notebook will not start in Sandbox 2.4

miranda_dhuitte — Mon, 30 May 2016 16:36:35 GMT

Thank you so much for your answer I had the same problem this morning and I thought it was a proxy problem but actually I had an empty note in Zeppelin. I hope this tiny problem will be fixed in the next versions of Zeppelin Notebook.

When I check the status of zeppelin-daemon.sh it tells me "Zeppelin running but process is dead" can anyone knows the reason for this error ?