Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

After Ambari 2.5.0.3 upgrade from 2.4 oozie server is not starting/stopping from Ambari ui

Highlighted

After Ambari 2.5.0.3 upgrade from 2.4 oozie server is not starting/stopping from Ambari ui

New Contributor

After ambari upgrade from 2.4 to 2.5.0.3 the oozie server is not starting,stopping or restarting from Ambari ui .I am able to do the same from commandline .

stderr: /var/lib/ambari-agent/data/errors-566.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 222, in <module>
    OozieServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 727, in restart
    self.stop(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 96, in stop
    oozie_service(action='stop', upgrade_type=upgrade_type)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py", line 181, in oozie_service
    user = params.oozie_user)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozied.sh stop 60 -force' returned 1. Setting OOZIE_HOME:          /usr/hdp/2.5.0.0-1245/oozie
Sourcing:                    /usr/hdp/2.5.0.0-1245/oozie/bin/oozie-env.sh
  setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-client/conf}
  setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-client/oozie-server}
  setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
  setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
  setting JAVA_HOME=/usr/java/default
  setting JRE_HOME=${JAVA_HOME}
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m -XX:MaxPermSize=256m"
  setting OOZIE_LOG=/var/log/oozie
  setting CATALINA_PID=/var/run/oozie/oozie.pid
  setting OOZIE_DATA=/hadoop/oozie/data
  setting OOZIE_HTTP_PORT=11000
  setting OOZIE_ADMIN_PORT=11001
  setting JAVA_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
  setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 "
Using   OOZIE_CONFIG:        /usr/hdp/current/oozie-server/conf
Sourcing:                    /usr/hdp/current/oozie-server/conf/oozie-env.sh
  setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-client/conf}
  setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-client/oozie-server}
  setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
  setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
  setting JAVA_HOME=/usr/java/default
  setting JRE_HOME=${JAVA_HOME}
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m -XX:MaxPermSize=256m"
  setting OOZIE_LOG=/var/log/oozie
  setting CATALINA_PID=/var/run/oozie/oozie.pid
  setting OOZIE_DATA=/hadoop/oozie/data
  setting OOZIE_HTTP_PORT=11000
  setting OOZIE_ADMIN_PORT=11001
  setting JAVA_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
  setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 "
Setting OOZIE_CONFIG_FILE:   oozie-site.xml
Using   OOZIE_DATA:          /hadoop/oozie/data
Using   OOZIE_LOG:           /var/log/oozie
Setting OOZIE_LOG4J_FILE:    oozie-log4j.properties
Setting OOZIE_LOG4J_RELOAD:  10
Setting OOZIE_HTTP_HOSTNAME: bhwx22hwxworker2.cse-int-06.local
Using   OOZIE_HTTP_PORT:     11000
Using   OOZIE_ADMIN_PORT:     11001
Setting OOZIE_HTTPS_PORT:     11443
Setting OOZIE_BASE_URL:      http://bhwx22hwxworker2.cse-int-06.local:11000/oozie
Using   CATALINA_BASE:       /usr/hdp/current/oozie-client/oozie-server
Setting OOZIE_HTTPS_KEYSTORE_FILE:     /home/oozie/.keystore
Setting OOZIE_HTTPS_KEYSTORE_PASS:     password
Setting OOZIE_INSTANCE_ID:       bhwx22hwxworker2.cse-int-06.local
Setting CATALINA_OUT:        /var/log/oozie/catalina.out
Using   CATALINA_PID:        /var/run/oozie/oozie.pid

Using   CATALINA_OPTS:       -Dhdp.version=2.5.0.0-1245  -Xmx2048m -XX:MaxPermSize=256m -Xmx2048m -XX:MaxPermSize=256m -Dderby.stream.error.file=/var/log/oozie/derby.log
Adding to CATALINA_OPTS:     -Doozie.home.dir=/usr/hdp/2.5.0.0-1245/oozie -Doozie.config.dir=/usr/hdp/current/oozie-server/conf -Doozie.log.dir=/var/log/oozie -Doozie.data.dir=/hadoop/oozie/data -Doozie.instance.id=bhwx22hwxworker2.cse-int-06.local -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=bhwx22hwxworker2.cse-int-06.local -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://bhwx22hwxworker2.cse-int-06.local:11000/oozie -Doozie.https.keystore.file=/home/oozie/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
PID file found but no matching process was found. Stop aborted.

stdout: /var/lib/ambari-agent/data/output-566.txt

2018-07-25 10:50:05,194 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-07-25 10:50:05,300 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-1245, current_cluster_version=2.5.0.0-1245 -> 2.5.0.0-1245
2018-07-25 10:50:05,301 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-07-25 10:50:05,302 - Group['hadoop'] {}
2018-07-25 10:50:05,304 - Group['users'] {}
2018-07-25 10:50:05,304 - Group['spark'] {}
2018-07-25 10:50:05,304 - Group['livy'] {}
2018-07-25 10:50:05,304 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,306 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2018-07-25 10:50:05,308 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2018-07-25 10:50:05,309 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,311 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,312 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,313 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,314 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2018-07-25 10:50:05,315 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,315 - User['gpadmin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,317 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,317 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,318 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,319 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2018-07-25 10:50:05,320 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-07-25 10:50:05,321 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-07-25 10:50:05,330 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-07-25 10:50:05,330 - Group['hdfs'] {}
2018-07-25 10:50:05,331 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2018-07-25 10:50:05,331 - FS Type: 
2018-07-25 10:50:05,332 - Directory['/etc/hadoop'] {'mode': 0755}
2018-07-25 10:50:05,346 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-07-25 10:50:05,346 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-07-25 10:50:05,360 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-07-25 10:50:05,370 - Skipping Execute[('setenforce', '0')] due to not_if
2018-07-25 10:50:05,370 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-07-25 10:50:05,372 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-07-25 10:50:05,373 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-07-25 10:50:05,373 - File['/var/lib/ambari-agent/lib/fast-hdfs-resource.jar'] {'content': StaticFile('fast-hdfs-resource.jar'), 'mode': 0644}
2018-07-25 10:50:05,427 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-07-25 10:50:05,429 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-07-25 10:50:05,433 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-07-25 10:50:05,441 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2018-07-25 10:50:05,442 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-07-25 10:50:05,443 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-07-25 10:50:05,446 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2018-07-25 10:50:05,454 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-07-25 10:50:05,671 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-1245, current_cluster_version=2.5.0.0-1245 -> 2.5.0.0-1245
2018-07-25 10:50:05,672 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-07-25 10:50:05,678 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1}
2018-07-25 10:50:05,714 - checked_call returned (0, '2.5.0.0-1245', '')
2018-07-25 10:50:05,717 - Directory['/var/tmp/oozie'] {'owner': 'oozie', 'create_parents': True}
2018-07-25 10:50:05,718 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozied.sh stop 60 -force'] {'environment': {'OOZIE_CONFIG': '/usr/hdp/current/oozie-server/conf'}, 'only_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'user': 'oozie'}
2018-07-25 10:50:05,838 - Execute['find /var/log/oozie -maxdepth 1 -type f -name '*' -exec echo '==> {} <==' \; -exec tail -n 40 {} \;'] {'logoutput': True, 'ignore_failures': True, 'user': 'oozie'}
==> /var/log/oozie/oozie.log-2018-07-24-02 <==
1 REPLY 1

Re: After Ambari 2.5.0.3 upgrade from 2.4 oozie server is not starting/stopping from Ambari ui

@tauqeer khan

delete this pid file form oozie machine and see if the issue resolves.

/var/run/oozie/oozie.pid
Don't have an account?
Coming from Hortonworks? Activate your account here