Support Questions

Find answers, ask questions, and share your expertise

ranger ssl does not start embedded tomcat server

avatar
Expert Contributor

Hi

We have gone through the entire process of automating our cluster using blueprints and have had several successful deployments using wild cart certs in all our environments. We recently hit a snag in one of our larger environments where the ranger-admin though successfully installed with no errors whatsoever does not init the embedded tomcat server to listen on port 6182 when configured for ssl.

on a similar environment this is from the catalina.out on /var/log/ranger/admin

Apr 19, 2018 6:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp] .....
Apr 19, 2018 6:02:07 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Apr 19, 2018 6:02:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp].
Apr 19, 2018 6:02:08 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-6182"]
Apr 19, 2018 6:02:08 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat
Apr 19, 2018 6:02:08 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.81
Apr 19, 2018 6:02:08 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class


but on the environment on which we have the problem the logs are as such

ava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
log4j:WARN No appenders could be found for logger (org.apache.tomcat.util.IntrospectionUtils).
log4j:WARN Please initialize the log4j system properly.log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Apr 19, 2018 6:09:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp
Apr 19, 2018 6:09:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp, webAppName = /
Apr 19, 2018 6:09:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp] .....
Apr 19, 2018 6:09:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp].
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
Apr 19, 2018 6:10:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 19, 2018 6:10:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST

Note the missing logs

Apr 19, 2018 6:02:08 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-6182"]
Apr 19, 2018 6:02:08 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat

The environments are similar in setup and there is no difference. Tried to enable debugging by setting debug level for apache.ranger and spring.frameworks in log4j.xml and from ranger-admin-log4j.xml in the ui. Still there were no errors. Tomcat does not listen on port 6182 and the work directory under /usr/hdp/.../ranger/admin/ews/ ...doesnt have anything ?

Any suggestions on how to further debug this ? (apart from removing the service and re-installing)

from the xa_portal.log it looks like the spring application context gets initialized ..which is wierd .. all service install but can't connect to ranger on port 6182 because it is not listening on port 6182 !

1 ACCEPTED SOLUTION

avatar
Expert Contributor

When in doubt; doubt SSL.

After several errors and trials, the core issue was that the alias in keystore for ranger was incorrect. What was really surprising that tomcat did not throw any errors whatsoever but just failed to start listening on port 6182. Increasing the debug level logs for several pacakges in the log4j for ranger-admin-env and even in /usd/hdp/current/..../ews/WEB-INF/ .... did inot show any error.

Usually we have seen errors in a normal tomcat ssl setup. It was very surprising that no error was thrown. The only error was that it did not boot up to listen on port 6182. strange.

View solution in original post

3 REPLIES 3

avatar

@Anshuman Mehta

Login to shell console on Ranger Admin host and as root user run:

# ps -ef | grep rangeradmin

# netstat -nap | grep <pid>

# grep -C2 https /data1/hdp/2.6.4.0-91/ranger-admin/conf/ranger-admin-site.xml

Paste/Attach the results here.

Also please perform an extended file list in:

ls -l /data1/hdp/2.6.4.0-91/ranger-admin/ews

avatar
Expert Contributor

@Felix Albani the formatting was off .. have cleaned it up. Thanks !

$ sudo ps -ef | grep rangeradminranger
2009 1 0 07:10 ? 00:01:42 java -Dproc_rangeradmin -XX:MaxPermSize=256m -Xmx1024m -Xms1024m -Duser.timezone=UTC -Dservername=rangeradmin -Dlogdir=/var/log/ranger/admin -Dcatalina.base=/data1/hdp/2.6.4.0-91/ranger-admin/ews -cp /data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp/WEB-INF/classes/conf:/data1/hdp/2.6.4.0-91/ranger-admin/ews/lib/*:/data1/hdp/2.6.4.0-91/ranger-admin/ews/ranger_jaas/*:/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp/WEB-INF/classes/conf/ranger_jaas:/usr/java/latest/lib/*:/*: org.apache.ranger.server.tomcat.EmbeddedServer
$ sudo netstat -anp | grep 2009
tcp 0 0 127.0.0.1:6085 0.0.0.0:* LISTEN 2009/java
tcp 0 0 10.108.10.112:44131 10.128.30.110:5432 ESTABLISHED 2009/java
tcp 0 0 10.108.10.112:44132 10.128.30.110:5432 ESTABLISHED 2009/java
tcp 0 0 10.108.10.112:44130 10.128.30.110:5432 ESTABLISHED 2009/java
tcp 0 0 10.108.10.112:44138 10.128.30.110:5432 ESTABLISHED 2009/java
tcp 0 0 10.108.10.112:44139 10.128.30.110:5432 ESTABLISHED 2009/java
unix 2 [ ] STREAM CONNECTED 9339763 2009/java</property>
$ grep https -C2 /data1/hdp/2.6.4.0-91/ranger-admin/conf/ranger-admin-site.xml
<property>
<name>ranger.externalurl</name>
<value>https://myserver:6182</value>
</property>
<property>
<name>ranger.https.attrib.keystore.file</name>
<value>/path/to/key/keystore.jks</value>
</property>
<property>
<name>ranger.service.https.attrib.client.auth</name>
<value>want</value>
</property>
<property>
<name>ranger.service.https.attrib.clientAuth</name>
<value>want</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.credential.alias</name>
<value>keyStoreCredentialAlias</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.file</name>
<value>/path/to/key/keystore.jks</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.keyalias</name>
<value>my_wildcard_alias</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.pass</name>
<value>_</value>
</property>
<property>
<name>ranger.service.https.attrib.ssl.enabled</name>
<value>true</value>
</property>
<property>
<name>ranger.service.https.port</name>
<value>6182</value>
</property>
<br>
$ ls -l /data1/hdp/2.6.4.0-91/ranger-admin/ews/
total 36
drwxr-xr-x 2 ranger ranger 4096 Apr 19 07:09 lib
lrwxrwxrwx 1 ranger ranger 21 Apr 19 00:23 logs -> /var/log/ranger/admin
-r-xr--r-- 1 ranger ranger 2192 Jan 4 10:47 ranger-admin-initd
-r-xr--r-- 1 ranger ranger 6347 Jan 4 10:47 ranger-admin-services.sh
lrwxrwxrwx 1 ranger ranger 58 Apr 19 00:23 ranger-admin-start -> /usr/hdp/2.6.4.0-91/ranger-admin/ews/start-ranger-admin.sh
lrwxrwxrwx 1 ranger ranger 57 Apr 19 00:23 ranger-admin-stop -> /usr/hdp/2.6.4.0-91/ranger-admin/ews/stop-ranger-admin.sh
drwxr-xr-x 2 ranger ranger 4096 Apr 19 00:23 ranger_jaas
-r-xr--r-- 1 ranger ranger 971 Jan 4 10:47 start-ranger-admin.sh
-r-xr--r-- 1 ranger ranger 969 Jan 4 10:47 stop-ranger-admin.sh
drwxr-xr-x 10 ranger ranger 4096 Apr 19 00:24 webapp
drwxr-xr-x 3 ranger ranger 4096 Apr 19 00:33 work
<br>

avatar
Expert Contributor

When in doubt; doubt SSL.

After several errors and trials, the core issue was that the alias in keystore for ranger was incorrect. What was really surprising that tomcat did not throw any errors whatsoever but just failed to start listening on port 6182. Increasing the debug level logs for several pacakges in the log4j for ranger-admin-env and even in /usd/hdp/current/..../ews/WEB-INF/ .... did inot show any error.

Usually we have seen errors in a normal tomcat ssl setup. It was very surprising that no error was thrown. The only error was that it did not boot up to listen on port 6182. strange.