Kafka Ranger SSL integration issue

[#1]

We are trying to enable SSL connection between Apache Ranger and Kafka cluster. After creating keystore and truststore for both Kafka and Ranger, we are unable to connect Kafka to Ranger and we are getting the following error message:

[2020-06-25 20:47:40,013] ERROR Unable to get the Credential Provider from the Configuration (org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider)
java.lang.IllegalArgumentException: The value of property hadoop.security.credential.provider.path must not be null
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1115)
at org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialProviders(RangerCredentialProvider.java:68)
at org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialString(RangerCredentialProvider.java:46)
at org.apache.ranger.plugin.util.RangerRESTClient.getCredential(RangerRESTClient.java:386)
at org.apache.ranger.plugin.util.RangerRESTClient.getKeyManagers(RangerRESTClient.java:272)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:188)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:176)
at org.apache.ranger.plugin.util.RangerRESTClient.getResource(RangerRESTClient.java:156)
at org.apache.ranger.admin.client.RangerAdminRESTClient.createWebResource(RangerAdminRESTClient.java:275)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)


[2020-06-25 20:47:40,013] ERROR PolicyRefresher(serviceName=KafkaTest): failed to refresh policies. Will continue to use last known version of policies (51) (org.apache.ranger.plugin.util.PolicyRefresher)
java.lang.IllegalArgumentException: TrustManager is not specified
at org.apache.commons.lang.Validate.notNull(Validate.java:192)
at org.apache.ranger.plugin.util.RangerRESTClient.getSSLContext(RangerRESTClient.java:369)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:190)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:176)
at org.apache.ranger.plugin.util.RangerRESTClient.getResource(RangerRESTClient.java:156)
at org.apache.ranger.admin.client.RangerAdminRESTClient.createWebResource(RangerAdminRESTClient.java:275)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)

[#2]

@SKL 

What are the values set for following properties?

xasecure.policymgr.clientssl.keystore.credential.file	
xasecure.policymgr.clientssl.truststore.credential.file

xasecure.policymgr.clientssl.truststore

1. Make sure that for first two properties have values in following format

jceks://file/<filename> or jceks://file/{{credential_file}}

     2. Make truststore is set in for the third property. As per the error it does not have any value.

Attaching a screenshot for reference.

[#3]

@ajinkyapatil 

Please see the properties which you asked for. I am still getting the same error message.

Please let me know if anything is missing here.

 <property>
<name>xasecure.policymgr.clientssl.keystore</name>
<value>/etc/hadoop/conf/kafka.admin.keystore.jks</value>
<description>
Java Keystore files
</description>
</property>
 <property>
                <name>xasecure.policymgr.clientssl.keystore.password</name>
                <value>password</value>
        </property>
<property>
<name>xasecure.policymgr.clientssl.truststore</name>
<value>/etc/hadoop/conf/kafka.admin.truststore.jks</value>
<description>
java truststore file
</description>
</property>
 <property>
                <name>xasecure.policymgr.clientssl.truststore.password</name>
                <value>password</value>
        </property>
    <property>
<name>xasecure.policymgr.clientssl.keystore.credential.file</name>
<value>jceks://file/etc/ranger/KafkaTest/cred.jceks</value>
<description>
java  keystore credential file
</description>
</property>
<property>
<name>xasecure.policymgr.clientssl.truststore.credential.file</name>
<value>jceks://file/etc/ranger/KafkaTest/cred.jceks</value>
<description>
java  truststore credential file
</description>
</property>
</configuration>

[#4]

@ajinkyapatil 

We have two observations -

First observation

Even if we give a file-path which doesn't exist for Keystore & Trustsrore, Kafka Ranger (enable-kafka-plugin.sh)script is not complaining and going ahead with the generation of cred file. But eventually, Kafka is unable to connect to Ranger with the same error defined in the issue. 



Second observation

We are not sure but even after providing the right path for Keystore & Trustsrore, Kafka-Ranger (enable-kafka-plugin.sh) script seems to not care about the location of Keystore & Truststore and generate a cred file.

Attaching few screen prints

[#5]

Hi,

we have the same issue. How did you resolved this?

BR,

Gabor

[#6]

Hello @BGabor 

This error "TrustManager is not specified" can be thrown due to some of the following issues - 

Missing cert files or missing/wrong values for the below configs 

Make sure following properties are set:-

xasecure.policymgr.clientssl.keystore.credential.file=jceks://file/{{credential_file}}
xasecure.policymgr.clientssl.truststore.credential.file=jceks://file/{{credential_file}}
xasecure.policymgr.clientssl.truststore=/path/to/truststore

Also came across these Ranger jiras which indicates that the truststore info not specified in cacert or cacert needs to be manually configured. Note - they are fixed in Ranger 2.0.1, so you may also want to check the ranger version.

https://issues.apache.org/jira/browse/RANGER-2611

https://issues.apache.org/jira/browse/RANGER-2907