I faced a issue oozie with HA availabilty , loadbalancer, I found is a problem of java tomcat the jar files not loaded I did before a upgrade of jdbc driver manually , I provide you guys the log might someone drop any clue. Thanks

[#1]

java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
   at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:124)
   at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:52)
May 12, 2016 5:04:48 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
May 12, 2016 5:04:48 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-11000
May 12, 2016 5:04:48 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 496 ms
May 12, 2016 5:04:48 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
May 12, 2016 5:04:48 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.44
May 12, 2016 5:04:48 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor oozie.xml
May 12, 2016 5:04:48 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5-6.1.14.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
May 12, 2016 5:04:48 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
May 12, 2016 5:04:53 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory docs
May 12, 2016 5:04:53 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-11000
May 12, 2016 5:04:53 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 4693 ms
May 12, 2016 5:08:12 PM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-11000
May 12, 2016 5:08:13 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/oozie] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [FileWatchdog] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [main-SendThread(hc3.bmeu.com:2181)] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [ZKLocksChildReaper-0] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [IPC Client (1586289269) connection to hc2.bmeu.com/144.76.33.240:8020 from oozie/hc1.bmeu.com@TCAD.BMEU.COM] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [IPC Parameter Sending Thread #2] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.oozie.util.XLog$Info$1] (value [org.apache.oozie.util.XLog$Info$1@21a02556]) and a value of type [org.apache.oozie.util.XLog.Info] (value [org.apache.oozie.util.XLog$Info@3c8a7e38]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.hadoop.hdfs.DFSUtil$1] (value [org.apache.hadoop.hdfs.DFSUtil$1@138aa3cc]) and a value of type [java.util.Random] (value [java.util.Random@10a98392]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.hadoop.io.Text$1] (value [org.apache.hadoop.io.Text$1@30839e44]) and a value of type [sun.nio.cs.UTF_8.Encoder] (value [sun.nio.cs.UTF_8$Encoder@5f174dd2]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/oozie] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@45aca496]) and a value of type [org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData] (value [0 bytes read, 0 bytes written, 20 read ops, 0 large read ops, 0 write ops]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 12, 2016 5:08:14 PM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-11000
May 12, 2016 5:08:14 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.apache.hadoop.util.ShutdownHookManager$2.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
   at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:124)
   at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:52)
May 12, 2016 5:08:20 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
May 12, 2016 5:08:20 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-11000
May 12, 2016 5:08:20 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 368 ms
May 12, 2016 5:08:20 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
May 12, 2016 5:08:20 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.44
May 12, 2016 5:08:20 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor oozie.xml
May 12, 2016 5:08:20 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5-6.1.14.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
May 12, 2016 5:08:20 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
May 12, 2016 5:08:25 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory docs
May 12, 2016 5:08:25 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-11000
May 12, 2016 5:08:25 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 4449 ms
May 12, 2016 5:09:54 PM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-11000
May 12, 2016 5:09:55 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/oozie] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [FileWatchdog] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [main-SendThread(hc2.bmeu.com:2181)] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [ZKLocksChildReaper-0] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/oozie] appears to have started a thread named [IPC Parameter Sending Thread #0] but has failed to stop it. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.oozie.util.XLog$Info$1] (value [org.apache.oozie.util.XLog$Info$1@13dbed9e]) and a value of type [org.apache.oozie.util.XLog.Info] (value [org.apache.oozie.util.XLog$Info@67531e3a]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.hadoop.hdfs.DFSUtil$1] (value [org.apache.hadoop.hdfs.DFSUtil$1@2b38b1f]) and a value of type [java.util.Random] (value [java.util.Random@1d50a7ca]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.hadoop.io.Text$1] (value [org.apache.hadoop.io.Text$1@51ab1ee3]) and a value of type [sun.nio.cs.UTF_8.Encoder] (value [sun.nio.cs.UTF_8$Encoder@2e766822]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/oozie] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@28757abd]) and a value of type [org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData] (value [0 bytes read, 0 bytes written, 20 read ops, 0 large read ops, 0 write ops]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 12, 2016 5:09:55 PM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-11000
May 12, 2016 5:10:02 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
May 12, 2016 5:10:02 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-11000
May 12, 2016 5:10:02 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 378 ms
May 12, 2016 5:10:02 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
May 12, 2016 5:10:02 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.44
May 12, 2016 5:10:02 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor oozie.xml
May 12, 2016 5:10:02 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5-6.1.14.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class

[#2 · Accepted solution]

@Hamid Zorgani

1. Please add new updated jdbc driver jar file in /usr/hdp/current/oozie-server/libext/ and follow below steps to prepare new war file.

2. Stop oozie server

3. Run below command

/usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war 

4. Start Oozie server

If above solution does not work then:

Find and remove older jdbc driver from below location

/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/

And repeat Steps 1-4.

[#3]

Thanks for your help, now oozie two instance working but when I change the base uri : myloadbalncer I got 2 errors from both instances of oozie .

Connection failed to http://myloadbalancer.com:21000 (Execution of 'curl -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/3a24ff03-dd0c-46a2-a2f5-c2f96e59c55c -c /var/lib/ambari-agent/tmp/cookies/3a24ff03-dd0c-46a2-a2f5-c2f96e59c55c -w '%{http_code}' http://hc3.bmeu.com:21000 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpjp060g 2>/tmp/tmpv1DMX1' returned 7. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to hc3.loadbalancer com:21000; Connection refused 00

CRIT for 2 hours