Oozie services are not accessible

[#1]

Hi,

Oozie services are running but i'm unable to access it. After performing service check it says

IndexError: list index out of range

[#2]

@Hitesh Rajpurohit

Can you please explain more about your issue. Are you getting this error while accessing Oozie dashboard in HUE? or somewhere else.

Logs/Screenshots would be helpful.

[#3]

Hi kuldeep,

here are the log files

stderr: /var/lib/ambari-agent/data/errors-694.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 145, in <module>
    OozieServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 61, in service_check
    OozieServiceCheckDefault.oozie_smoke_shell_file(smoke_test_file_name, prepare_hdfs_file_name)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 77, in oozie_smoke_shell_file
    oozie_examples_dir = glob.glob(params.oozie_examples_regex)[0]
IndexError: list index out of range

stdout: /var/lib/ambari-agent/data/output-694.txt

2017-01-31 12:00:40,402 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-01-31 12:00:40,440 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1}
2017-01-31 12:00:41,024 - checked_call returned (0, '2.5.0.0-1245', '')
2017-01-31 12:00:41,037 - XmlConfig['yarn-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644, 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configurations': ...}
2017-01-31 12:00:41,133 - Generating config: /usr/hdp/current/hadoop-client/conf/yarn-site.xml
2017-01-31 12:00:41,134 - File['/usr/hdp/current/hadoop-client/conf/yarn-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-01-31 12:00:41,496 - File['/var/lib/ambari-agent/tmp/oozieSmoke2.sh'] {'content': StaticFile('oozieSmoke2.sh'), 'mode': 0755}
2017-01-31 12:00:41,522 - File['/var/lib/ambari-agent/tmp/prepareOozieHdfsDirectories.sh'] {'content': StaticFile('prepareOozieHdfsDirectories.sh'), 'mode': 0755}

Command failed after 1 tries

[#4]

@Hitesh Rajpurohit

One reason when that can happen is If the following directories does not.

/usr/hdp/current/oozie-client/doc/
/usr/hdp/current/oozie-server/doc/

Ambari calculates the value of "oozie_examples_regex" as following:

oozie_examples_regex = format("{stack_root}/current/{oozie_root}/doc")


Else default value is following but for HDP stack it should be above one:
oozie_examples_regex = "/usr/share/doc/oozie-*"

So may be the following "/usr/hdp/current/oozie-server/doc/" (OR) /usr/hdp/current/oozie-client/doc/ directory might not be existing.

In later version of Ambari (3.0) there are few enhancements happened to make the Error message more meaningful and descriptive in such cases. You can find the fix here:

https://git1-us-west.apache.org/repos/asf?p=ambari.git;a=commitdiff;h=41c49e162f156320a0c366f487beea...

.

Also If the "doc" directory does not exist in your installation then one Possible reason is that "/etc/yum.conf" contains" tsflags=nodocs which prevents this folder from being installed along with oozie-client package. If this is the case, please fix /etc/yum.conf and re-install the package.

.

[#5]

Hi Jay,

After creating those directories i'm getting this error now

stderr: /var/lib/ambari-agent/data/errors-696.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 145, in <module>
    OozieServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 61, in service_check
    OozieServiceCheckDefault.oozie_smoke_shell_file(smoke_test_file_name, prepare_hdfs_file_name)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 82, in oozie_smoke_shell_file
    logoutput=True
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/var/lib/ambari-agent/tmp/prepareOozieHdfsDirectories.sh /usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf sandbox.hortonworks.com:8050 hdfs://sandbox.hortonworks.com:8020 default' returned 2. tar (child): oozie-examples.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
sed: can't read examples/apps/map-reduce/job.properties: No such file or directory
sed: can't read examples/apps/map-reduce/job.properties: No such file or directory
sed: can't read examples/apps/map-reduce/job.properties: No such file or directory
sed: can't read examples/apps/map-reduce/job.properties: No such file or directory
sed: can't read examples/apps/map-reduce/job.properties: No such file or directory
sed: can't read examples/apps/map-reduce/job.properties: No such file or directory
sed: can't read examples/apps/map-reduce/job.properties: No such file or directory

[#6]

@Hitesh Rajpurohit

Looks like you misunderstood what i said. "Creating the following directory only will not work".

/usr/hdp/current/oozie-client/doc/
/usr/hdp/current/oozie-server/doc/

.

The above directories should contain various sub directories and "oozie-examples.tar.gz" as well.

For example the mentioned directory should contain the following contents:

# ls -lart /usr/hdp/current/oozie-client/doc/
total 216
-rw-r--r--.  1 root root 106805 Nov 30 04:16 release-log.txt
-rw-r--r--.  1 root root   3167 Nov 30 04:16 README.txt
-rwxr-xr-x.  1 root root  45344 Nov 30 04:16 oozie-examples.tar.gz
-rw-r--r--.  1 root root    909 Nov 30 04:16 NOTICE.txt
-rw-r--r--.  1 root root  37664 Nov 30 04:16 LICENSE.txt
-rw-r--r--.  1 root root   1366 Nov 30 04:16 configuration.xsl

.

- For testing i would suggest you to reinstall the "oozie-client" on the host where the Service check got executed. And make sure that this time "/etc/yum.conf" does not contain "tsflags=nodocs". Sothat the doc also gets installed (not skipped)

[#7]

@Hitesh Rajpurohit

The Oozie service check will try to extract the "/usr/hdp/current/oozie-client/doc/oozie-examples.tar.gz" file. This file contains the following and additional files. So the host in which the service check is running must have that file

"/usr/hdp/current/oozie-client/doc/oozie-examples.tar.gz"

"examples/apps/map-reduce/job.properties"