Support Questions
Find answers, ask questions, and share your expertise

Zeppelin Notebook Service Check Fails During Upgrade

Zeppelin Notebook Service Check Fails During Upgrade

New Contributor

I'm running Ambari 2.6.1.0, and am attempting to upgrade a cluster from HDP-2.6.2.0 to HDP 2.6.4.0. During the upgrade, the Zeppelin Notebook service check fails at the following stage:

upgrade-dialog.png

The error message is as follows:

2018-03-09 10:22:41,874 - call['ambari-python-wrap /usr/bin/hdp-select status spark-client'] {'timeout': 20}
2018-03-09 10:22:41,906 - call returned (0, 'spark-client - 2.6.4.0-91')
2018-03-09 10:22:41,924 - Using hadoop conf dir: /usr/hdp/2.6.4.0-91/hadoop/conf
2018-03-09
 10:22:41,925 - Execute['curl -s -o /dev/null -w'%{http_code}' 
--negotiate -u: -k https://zeppelin.some.domain:9995 | grep 200'] 
{'logoutput': True, 'tries': 10, 'try_sleep': 3}
2018-03-09 
10:22:41,947 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1. 
2018-03-09
 10:22:44,966 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1. 
2018-03-09
 10:22:47,986 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1. 
2018-03-09
 10:22:51,005 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1. 
2018-03-09
 10:22:54,024 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1. 
2018-03-09
 10:22:57,043 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1. 
2018-03-09
 10:23:00,066 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1. 
2018-03-09
 10:23:03,085 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1. 
2018-03-09
 10:23:06,106 - Retrying after 3 seconds. Reason: Execution of 'curl -s 
-o /dev/null -w'%{http_code}' --negotiate -u: -k 
https://zeppelin.some.domain:9995 | grep 200' returned 1.

The cluster that is being upgraded does not use Kerberos, and Zeppelin is configured to use LDAP authentication. Manually running the relevant curl command (sans null output) returned the following, which seems appropriate under the circumstances:

<!doctype html><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--> <html ng-controller="MainCtrl" class="no-js"> <head> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta charset="utf-8"> <title></title> <!-- disable caches for all browser --> <meta http-equiv="cache-control" content="max-age=0"> <meta http-equiv="cache-control" content="no-cache"> <meta http-equiv="cache-control" content="no-store"> <meta http-equiv="expires" content="0"> <meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT"> <meta http-equiv="pragma" content="no-cache"> <meta name="description" content=""> <meta name="viewport" content="width=device-width"> <!-- ACE CSS autocomplition issue workarounds --> <style type="text/css">div.ace_editor.ace_autocomplete .ace_marker-layer .ace_active-line { z-index: -1 !important; }</style> <!-- Place favicon.ico and apple-touch-icon.png in the root directory --> <link rel="stylesheet" href="styles/vendor.ead2f1215177b914.css"> <link rel="stylesheet" href="styles/main.5e840e18559d1996.css"> <link rel="stylesheet" ng-href="assets/styles/looknfeel/{{looknfeel}}.css"> <link rel="stylesheet" href="assets/styles/printMode.css">  <body ng-class="{'bodyAsIframe': asIframe}"> <!--[if lt IE 7]>
    <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
    <![endif]--> <toast></toast> <div ng-include src="'components/navbar/navbar.html?v=1515063488639'"></div> <!-- Add your site or application content here --> <div id="main" class="container"> <div ng-view></div> </div> <!-- Modal ::  Keyboard shortcuts --> <div ng-include src="'components/modal-shortcut/modal-shortcut.html?v=1515063488639'"></div> <div ng-controller="NotenameCtrl as notenamectrl"> <div id="note-modal-container" ng-include src="'components/noteName-create/note-name-dialog.html?v=1515063488639'"></div> </div> <div ng-controller="NoteImportCtrl as noteimportctrl"> <div id="note-import-container" ng-include src="'components/noteName-import/note-import-dialog.html?v=1515063488639'"></div> </div> <div ng-controller="LoginCtrl as noteimportctrl"> <div id="login-container" ng-include src="'components/login/login.html?v=1515063488639'"></div> </div> <div ng-controller="RenameCtrl"> <div ng-include src="'components/rename/rename.html?v=1515063488639'"></div> </div> <!--[if lt IE 9]>
    <script src="scripts/oldieshim.8967257085f0b83a.js"></script>
    <![endif]--> <script type="text/x-mathjax-config">var config = {
        extensions: ["tex2jax.js"],
        jax: ["input/TeX", "output/HTML-CSS"],
        tex2jax: {
          inlineMath: [ ["\\(","\\)"] ],
          displayMath: [ ['$$','$$'], ["\\[","\\]"] ],
          processEscapes: true
        },
        "HTML-CSS": { availableFonts: ["TeX"] },
        messageStyle: "none"
      }
      // add root only if it's not dev mode
      if (Number(location.port) !== 9000) {
        config.root = '.';
      }
      MathJax.Hub.Config(config);</script> <script src="scripts/vendor.b8e51cdeeccd208d.js"></script> <script type="text/javascript" src="app.05bbdae750681c30f521.js"></script>

To try and elicit a different response during the upgrade's service check process, I paused the upgrade installation, stopped the Zeppelin service and then resumed the upgrade and re-ran the check. However, even though the curl response code returned was 000 (as might be expected, given that the service was down), the check still failed with the same message.

Of note, as part of the upgrade process I performed the migration to HDFS of the notebooks and interpreters that were formerly resident on the Zeppelin server, as documented here:

Apache Zeppelin Component Guide

I am able to log onto the Zeppelin server via LDAP, and the notebooks and interpreters are present.

Has anyone encountered this before?