Created 05-12-2016 08:56 PM
java.lang.IllegalStateException at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526) at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:124) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:52) May 12, 2016 5:04:48 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 May 12, 2016 5:04:48 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-11000 May 12, 2016 5:04:48 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 496 ms May 12, 2016 5:04:48 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina May 12, 2016 5:04:48 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.44 May 12, 2016 5:04:48 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor oozie.xml May 12, 2016 5:04:48 PM org.apache.catalina.loader.WebappClassLoader validateJarFile INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5-6.1.14.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class May 12, 2016 5:04:48 PM org.apache.catalina.loader.WebappClassLoader validateJarFile INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class May 12, 2016 5:04:53 PM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory docs May 12, 2016 5:04:53 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-11000 May 12, 2016 5:04:53 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 4693 ms May 12, 2016 5:08:12 PM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-11000 May 12, 2016 5:08:13 PM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/oozie] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [FileWatchdog] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [main-SendThread(hc3.bmeu.com:2181)] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [ZKLocksChildReaper-0] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [IPC Client (1586289269) connection to hc2.bmeu.com/144.76.33.240:8020 from oozie/hc1.bmeu.com@TCAD.BMEU.COM] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [IPC Parameter Sending Thread #2] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.oozie.util.XLog$Info$1] (value [org.apache.oozie.util.XLog$Info$1@21a02556]) and a value of type [org.apache.oozie.util.XLog.Info] (value [org.apache.oozie.util.XLog$Info@3c8a7e38]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.hadoop.hdfs.DFSUtil$1] (value [org.apache.hadoop.hdfs.DFSUtil$1@138aa3cc]) and a value of type [java.util.Random] (value [java.util.Random@10a98392]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.hadoop.io.Text$1] (value [org.apache.hadoop.io.Text$1@30839e44]) and a value of type [sun.nio.cs.UTF_8.Encoder] (value [sun.nio.cs.UTF_8$Encoder@5f174dd2]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. May 12, 2016 5:08:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/oozie] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@45aca496]) and a value of type [org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData] (value [0 bytes read, 0 bytes written, 20 read ops, 0 large read ops, 0 write ops]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. May 12, 2016 5:08:14 PM org.apache.coyote.http11.Http11Protocol destroy INFO: Stopping Coyote HTTP/1.1 on http-11000 May 12, 2016 5:08:14 PM org.apache.catalina.loader.WebappClassLoader loadClass INFO: Illegal access: this web application instance has been stopped already. Could not load org.apache.hadoop.util.ShutdownHookManager$2. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact. java.lang.IllegalStateException at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526) at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:124) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:52) May 12, 2016 5:08:20 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 May 12, 2016 5:08:20 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-11000 May 12, 2016 5:08:20 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 368 ms May 12, 2016 5:08:20 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina May 12, 2016 5:08:20 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.44 May 12, 2016 5:08:20 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor oozie.xml May 12, 2016 5:08:20 PM org.apache.catalina.loader.WebappClassLoader validateJarFile INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5-6.1.14.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class May 12, 2016 5:08:20 PM org.apache.catalina.loader.WebappClassLoader validateJarFile INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class May 12, 2016 5:08:25 PM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory docs May 12, 2016 5:08:25 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-11000 May 12, 2016 5:08:25 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 4449 ms May 12, 2016 5:09:54 PM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-11000 May 12, 2016 5:09:55 PM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/oozie] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [FileWatchdog] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [main-SendThread(hc2.bmeu.com:2181)] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [ZKLocksChildReaper-0] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [IPC Parameter Sending Thread #0] but has failed to stop it. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.oozie.util.XLog$Info$1] (value [org.apache.oozie.util.XLog$Info$1@13dbed9e]) and a value of type [org.apache.oozie.util.XLog.Info] (value [org.apache.oozie.util.XLog$Info@67531e3a]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.hadoop.hdfs.DFSUtil$1] (value [org.apache.hadoop.hdfs.DFSUtil$1@2b38b1f]) and a value of type [java.util.Random] (value [java.util.Random@1d50a7ca]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/oozie] created a ThreadLocal with key of type [org.apache.hadoop.io.Text$1] (value [org.apache.hadoop.io.Text$1@51ab1ee3]) and a value of type [sun.nio.cs.UTF_8.Encoder] (value [sun.nio.cs.UTF_8$Encoder@2e766822]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/oozie] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@28757abd]) and a value of type [org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData] (value [0 bytes read, 0 bytes written, 20 read ops, 0 large read ops, 0 write ops]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. May 12, 2016 5:09:55 PM org.apache.coyote.http11.Http11Protocol destroy INFO: Stopping Coyote HTTP/1.1 on http-11000 May 12, 2016 5:10:02 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 May 12, 2016 5:10:02 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-11000 May 12, 2016 5:10:02 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 378 ms May 12, 2016 5:10:02 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina May 12, 2016 5:10:02 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.44 May 12, 2016 5:10:02 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor oozie.xml May 12, 2016 5:10:02 PM org.apache.catalina.loader.WebappClassLoader validateJarFile INFO: validateJarFile(/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5-6.1.14.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Created 05-13-2016 06:33 PM
1. Please add new updated jdbc driver jar file in /usr/hdp/current/oozie-server/libext/ and follow below steps to prepare new war file.
2. Stop oozie server
3. Run below command
/usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war
4. Start Oozie server
If above solution does not work then:
Find and remove older jdbc driver from below location
/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/
And repeat Steps 1-4.
Created 05-13-2016 06:33 PM
1. Please add new updated jdbc driver jar file in /usr/hdp/current/oozie-server/libext/ and follow below steps to prepare new war file.
2. Stop oozie server
3. Run below command
/usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war
4. Start Oozie server
If above solution does not work then:
Find and remove older jdbc driver from below location
/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib/
And repeat Steps 1-4.
Created 05-13-2016 08:39 PM
Thanks for your help, now oozie two instance working but when I change the base uri : myloadbalncer I got 2 errors from both instances of oozie .
Connection failed to http://myloadbalancer.com:21000 (Execution of 'curl -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/3a24ff03-dd0c-46a2-a2f5-c2f96e59c55c -c /var/lib/ambari-agent/tmp/cookies/3a24ff03-dd0c-46a2-a2f5-c2f96e59c55c -w '%{http_code}' http://hc3.bmeu.com:21000 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpjp060g 2>/tmp/tmpv1DMX1' returned 7. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to hc3.loadbalancer com:21000; Connection refused 00
CRIT for 2 hours