CA signed cert for Ranger ; plugins don't show up

[#1]

Hi

I have a CA signed wildcard cert for my company like *.mycompany.com and am attempting to set it up for the cluster ssl setup. I have it setup successfully for all components except solr and ranger.

Specific to ranger my intention to use the CA signed cert and key for ALL the ranger plugins and the ranger admin . I understand that without kerberos there can only be 2 way ssl.

After following the steps as documented here ranger admin serves up properly however

1. during ranger admin client install the solr cloud cannot create the ranger-audit collection because the cert that it is trying to verify tries to pick up the ip instead of the hostname which i will try and follow up with this

2. None of the hdfs/hbase/hive plugins appear in the ranger admin and when I attempt to test connection in the kms view of ranger admin the test fails saying that keyadmin user has no authorization for "GET keys"

so my question is that will the above setup work i.e can i use the same keystore for all plugins and the ranger ui using the wildcard certificate and then use the same truststore for all ? we maintain our own network level security .

I am on the hdp 2.6.4 stack

[#2]

Any specific reason you are not using kerberos since that is the recommended config. In that case, 2 way SSL is not required.

Have you checked why plugins are not able to communicate with ranger admin? What errors do you see?

Make sure common name specified in HDFS/Hive/etc service (repository) in ranger admin UI matches CN.

[#3]

We are not on a kerberos environment yet.

in terms of errors

in /var/log/ranger/admin/xa_portal.log

2018-01-31 00:00:17,150 [http-bio-6182-exec-2] ERROR org.apache.ranger.common.ServiceUtil (ServiceUtil.java:1376) - Unauthorized access. Unable to get client certificate. serviceName=HadoopCluster_hbase
2018-01-31 00:00:17,151 [http-bio-6182-exec-2] INFO  org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:63) - Request failed. loginId=null, logMessage=Unauthorized access - unable to get client ce
rtificate
javax.ws.rs.WebApplicationException
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:325)
        at org.apache.ranger.common.ServiceUtil.isValidateHttpsAuthentication(ServiceUtil.java:1377)
        at org.apache.ranger.rest.ServiceREST.getServicePoliciesIfUpdated(ServiceREST.java:2567)
        at org.apache.ranger.rest.ServiceREST$$FastClassBySpringCGLIB$$92dab672.invoke(<generated>)

2018-01-31 00:00:17,151 [http-bio-6182-exec-2] INFO  org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:326) - Operation error. response=VXResponse={org.apache.ranger.view.VXResponse@2a28b481statu
sCode={1} msgDesc={Unauthorized access - unable to get client certificate} messageList={[VXMessage={org.apache.ranger.view.VXMessage@6f0ff521name={OPER_NOT_ALLOWED_FOR_ENTITY} rbKey={xa.error.oper_not_all
owed_for_state} message={Operation not allowed for entity} objectId={null} fieldName={null} }]} }
javax.ws.rs.WebApplicationException
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:325)
        at org.apache.ranger.common.ServiceUtil.isValidateHttpsAuthentication(ServiceUtil.java:1377)
        at org.apache.ranger.rest.ServiceREST.getServicePoliciesIfUpdated(ServiceREST.java:2567)
        at org.apache.ranger.rest.ServiceREST$$FastClassBySpringCGLIB$$92dab672.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

in /var/log/ranger/kms/kms.log

2018-01-31 00:00:17,544 ERROR PolicyRefresher - PolicyRefresher(serviceName=HadoopCluster_kms): failed to refresh policies. Will continue to use last known version of policies (-1)
java.lang.IllegalArgumentException: SSLContext must not be null
        at com.sun.jersey.client.urlconnection.HTTPSProperties.<init>(HTTPSProperties.java:106)
        at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.ja

(don't think this is used anywhere .. and can be ignored)
2018-01-31 00:00:17,529 WARN  FSInputChecker - Problem opening checksum file: file:/etc/ranger/HadoopCluster_kms/cred.jceks.  Ignoring exception: 
java.io.FileNotFoundException: /etc/ranger/HadoopCluster_kms/.cred.jceks.crc (Permission denied)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)

in /var/log/hadoop/hdfs/hadoop-hdfs-namednode

2018-01-31 18:17:05,296 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=false, user=hdfs (auth:SIMPLE), response={"httpStatusCode":400,"statusCode":0}, serviceName=HadoopCluster_hadoop
2018-01-31 18:17:06,824 INFO  BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1653)) - BLOCK* neededReplications = 0, pendingReplications = 0.
2018-01-31 18:17:08,325 WARN  mortbay.log (Slf4jLog.java:warn(76)) - SSL renegotiate denied: java.nio.channels.SocketChannel[connected local=/192.168.10.20:50470 remote=/192.168.10.20:45972]
2018-01-31 18:17:08,333 WARN  mortbay.log (Slf4jLog.java:warn(76)) - SSL renegotiate denied: java.nio.channels.SocketChannel[connected local=/192.168.10.20:50470 remote=/192.168.10.20:45970]

In the ranger admin UI logged in as the keyadmin user > service manager > edit kms service > test connection

rg.apache.ranger.plugin.client.HadoopException: {
"RemoteException" : {
"message" : "User:keyadmin not allowed to do 'GET_KEYS'",
"exception" : "AuthorizationException",
"javaClassName" : "org.apache.hadoop.security.authorize.AuthorizationException"
}
}. 
{
"RemoteException" : {
"message" : "User:keyadmin not allowed to do 'GET_KEYS'",
"exception" : "AuthorizationException",
"javaClassName" : "org.apache.hadoop.security.authorize.AuthorizationException"
}
}. 

[#4]

@vperiasamy am trying to understand what the relevance of the note at the bottom of this solution . Is that solution upto date ?

Note: while creating the client certs, make sure you provide extension as"usr_cert"and server cert as"server_cert", other wise 2 WAY SSL communication would fail.

[#5]

I checked my certificate and under Extended Key Usage it has both server Authentication and Client Authentication as a value.

[#6 · Accepted solution]

The root cause of the issue was that the intermediate AND the root certificates were not imported into the server keystores. Took a bit of debugging the source to figure it out but it worked in the end. There were a couple of hiccups in terms of what ambari blueprints automates in terms of policy configurations vs what it does not. Also need to ensure that commonNameForCertificate is set appropriately to the alias of the certificate.